Systems and methods for manipulating a virtual environment

ABSTRACT

One variation of a method for manipulating virtual objects within a virtual environment includes: receiving a touch image from a handheld device, the touch image comprising representations of discrete inputs into a touch sensor integrated into the handheld device; extracting a first force magnitude of a first input at a first location on a first side of the handheld device from the touch image; extracting a second force magnitude of a second input at a second location on a second side of the handheld device from the touch image, the second side of the handheld device opposite the first side of the handheld device; transforming the first input and the second input into a gesture; assigning a magnitude to the gesture based on the first force magnitude; and manipulating a virtual object within a virtual environment based on a type and the magnitude of the gesture.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit of U.S. Provisional Application No.62/198,222, filed on 29 Jul. 2015, which is incorporated in its entiretyby this reference.

This Application is related to U.S. patent application Ser. No.14/499,001, filed on 26 Sep. 2014, which is incorporated in its entiretyby this reference.

TECHNICAL FIELD

This invention relates generally to the field of touch sensors and morespecifically to a new and useful system and method for manipulating avirtual environment.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a first method;

FIG. 2 is a flowchart representation of one variation of the firstmethod;

FIG. 3 is a flowchart representation of one variation of the firstmethod and a second method;

FIG. 4 is a flowchart representation of one variation of the firstmethod and one variation of the second method;

FIG. 5 is a flowchart representation of one variation of the secondmethod;

FIG. 6 is a flowchart representation of a system;

FIG. 7 is a flowchart representation of one variation of the system;

FIG. 8 is a flowchart representation of one variation of the system;

FIG. 9 is a flowchart representation of one variation of the system; and

FIG. 10 is a schematic representation of one variation of the system.

DESCRIPTION OF THE EMBODIMENTS

The following description of the embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.

1. First Method

As shown in FIG. 1, a first method S100 for manipulating a virtualenvironment includes: detecting a touch sensor surface in a real spacein Block S110; based on a position of the touch sensor surface in thereal space, mapping the touch sensor surface to a virtual surface in avirtual environment in Block S120; receiving a touch image defining amagnitude and a position of a force on the touch sensor surface in BlockS130; based on an orientation of the touch sensor surface in the realspace, assigning a three-dimensional direction to the force to constructa vector in Block S140; and modifying the virtual surface according tothe vector within the virtual environment in Block S150.

As shown in FIG. 3, one variation of the first method S100 includes: ata first time, determining a first position of a touch sensor within areal space, the touch sensor including a touch sensor surface in BlockS110; based on the first position of the touch sensor within the realspace at the first time, bounding a virtual surface of a virtual objectwithin the virtual environment tractable through inputs across an areaof the touch sensor surface in Block S120; generating a first forcevector including a magnitude related to a force magnitude of a firstinput on the touch sensor surface and a direction related to anorientation of the touch sensor within the real space at the first timein Block S140; locating an origin of the first force vector within thevirtual environment based on a first location of the first input on thetouch sensor surface and the first position of the touch sensor withinthe real space in Block S142; and manipulating the virtual surface ofthe virtual object within the virtual environment according to the firstforce vector in Block S150.

1.1 Applications

Generally, the first method S100 can be executed by a computer systemhosting a virtual environment to link user inputs within physical spaceto virtual objects within the virtual environment. In particular, thecomputer system can interface with an input device (e.g., a handhelddevice) including a touch sensor surface to receive force magnitude andposition data of a user input on the touch sensor, track the positionand orientation of the input device within the real space, map the realtouch sensor to a virtual surface or virtual object within the virtualenvironment based on the position and orientation of the input device inreal space, and then manipulate the virtual surface or virtual objectbased on the force magnitude and location of the input on the touchsensor surface and the position and orientation of the input device inreal space.

A computer system executing the first method S100 can fuse the forcemagnitude of an input on a touch sensor surface, an orientation of thetouch sensor surface in real space, and the location of the input on thetouch sensor surface into: a force vector (i.e., a force magnitude and adirection); and an origin of the force vector within the virtualenvironment. The computer system can then project the force vector intoa virtual environment to define a magnitude and direction of a change inthe position, size, or geometry, etc. of a virtual object and/or avirtual surface within the virtual environment, thereby linking userinputs on the touch sensor surface in physical (i.e., real) space tocontrol or manipulate virtual objects or virtual surfaces within thevirtual environment.

In one example of the first method S100, in response to application ofan input object (e.g., a stylus, a finger) on a particular staticlocation on the touch sensor surface, wherein the particular location onthe touch sensor surface is mapped to an area on a virtual surface of avirtual three-dimensional object, the computer system can shift thisarea of the virtual three-dimensional object inward at a rate and/or bya degree corresponding to the force magnitude of the input and in adirection corresponding to the orientation of the touch sensor surfacein real space, as shown in FIG. 2. In this example, as the input objectmoves from the particular location to a second location on the touchsensor surface, the computer system can displace this area on thesurface of the virtual object in a direction corresponding to (oropposite) the initial magnitude of the applied force and form a virtualvalley in the surface of the virtual object along a trajectorycorresponding to the trajectory of the input object from the initialcontact location to the second location on the touch sensor surface,thereby enabling a user to move, manipulate, and modify a virtualsurface of a virtual object within a virtual environment by moving atouch sensor to a position in real space corresponding to the virtualsurface and then modulating the magnitude and position of a forceapplied to the touch sensor, such as with a finger or stylus.

The computer system can implement Blocks of the first method S100 (andthe second method S200 described below) to map a force applied to thetouch sensor surface to a virtual object or to a surface of a virtualobject, such as to push or pull (i.e., translate) the virtual object, torotate the virtual object, to bend or twist the virtual object, todeform the virtual surface, to smooth or texture the virtual surface,etc. within a virtual environment according to various input modes. Thefirst method S100 (and the second method S200 described below) isdescribed herein generally to deform or “push” a virtual object orvirtual surface inwardly according to a force applied to the touchsensor surface of the input device; however, the computer system canalso pull, twist, rotate, smooth, roughen (e.g., “texture”), orotherwise manipulate or modify a virtual object or virtual surfaceaccording to a force applied to the touch sensor surface. Furthermore,the computer system can selectively apply these modes during operation,such as: modally according to selection of a button or switch on theinput device; or gesturally according to a gesture applied to the touchsensor surface, a number of fingers applied to the touch sensor surfaceby a user, or a part of a hand or finger applied to the touch sensorsurface by the user, as described below.

The first method S100 is described herein as a method for manipulating avirtual environment, such as a virtual reality environment. However, thefirst method S100 can be similarly implemented to manipulate anaugmented reality environment, such as within a surgical environment.

1.2 Input Device

Block S130 of the first method S100 recites receiving a touch imagedefining a magnitude and a position of a force on the touch sensorsurface. Generally, in Block S130, the computer system interfaces with(or incorporates, integrates with) an input device including a touchsensor and a touch sensor surface, as described below, to receive aposition and force magnitude of an input on the touch sensor surface,such as in the form of a touch image.

The input device is described herein as an input device that includes atleast one pressure-sensitive touch sensor and that outputs the location((X,Y) position) and force magnitude of one or more contact areas (or“inputs”) on the (real) touch sensor surface. In one example, the inputdevice includes a tablet containing an opaque rectilinear body and aplanar touch sensor surface on one side of the opaque body. In a similarexample, the input device can include an opaque rectilinear bodydefining two opposing sides and a planar touch sensor surface acrosseach of the two sides. Alternatively, the input device can include aspherical, cubic, brick-shaped, or soap bar-shaped (e.g., rectilinearwith fileted corners) body with a touch sensor surface integrated acrossone or more planar or non-planar surfaces of the body.

Generally, the input device can incorporate one or morepressure-sensitive touch sensors including an array of sense electrodes(e.g., sensing pixels, or sense elements), such as described in U.S.patent application Ser. No. 14/499,001, and can output a single forcemagnitude value per contact area (i.e., an area of the touch sensorsurface contacted by a real object, such as a finger, a stylus, or apaintbrush) on the touch sensor surface per scan period. For example,the input device can scan the pressure-sensitive touch sensor at a rateof 100 Hz and record a peak force within a contact area detected on thetouch sensor surface, an average force across the contact area, and/or aforce at an effective center or a centroid of the contact area for eachdiscrete contact area on the touch sensor surface for each scan period.The touch sensor can then package these input locations, input areas,and input force magnitudes into one matrix or one other container perscan period before uploading this container to the computer system inBlock S130.

Alternatively, the input device can record a force value at each senseelement location within the pressure-sensitive touch sensor and packagethese force values into one touch image per scan period. For example,the input device can include: a grid array of discrete sense elements(e.g., sense and drive electrode pairs) arranged under a resistive layerexhibiting local variations in contact resistance with the sense elementresponsive to local changes in force applied to the touch sensor surfaceover the resistive layer; and a controller configured to read aresistance value from each sensor during a scan period, to transformeach resistance value into a force magnitude value, and to populate atouch image within these force values. The controller can alsointerpolate force magnitudes between adjacent sense element locations toachieve a touch image containing force magnitude values representingsense element locations at a resolution of one millimeter across thetouch sensor surface. The input device can thus detect and record theforce magnitude—over a spectrum of force magnitudes sensible by thearray of sense elements—of an input on the touch sensor, detectlocations of an input as the input moves over the touch sensor surface,and detect multiple inputs simultaneously across the touch sensorsurface.

Therefore, in addition to force magnitude of an input on the touchsensor surface, the input device can also output one or more locations(e.g., X and Y coordinates) on the touch sensor surface in contact witha real object. For example, the input device can output the (X,Y)position of single force value on the touch sensor surface or the (X,Y)position of each force value on the touch sensor surface above abaseline force value. In this example, the input device can reset abaseline force value for the pressure-sensitive touch sensor over time,such as based on the mode of force values across the touch sensorsurface to compensate for replacement of a physical overlay over thesurface, to compensate for changes in barometric pressure, and/or tocompensate for sensor drift across the pressure-sensitive touch sensor.The input device can thus generate and output a touch image containingsufficient information to reconstruct the position, shape, contactprofile, location, and applied force of an object in contact with thetouch sensor surface.

For each scan period, the input device can generate a touch image (e.g.,in the form of a matrix) representing force values (differential forcevalues, relative a baseline force value) measured at each discrete (andinterpolated) sensor location within the pressure-sensitive touchsensor, as shown in FIG. 1, and the input device can push (i.e.,transmit) the touch image for each scan period to the computer systemexecuting the first method S100 substantially in real-time over wired orwireless communication protocol in Block S130. Alternatively, thecomputer system can regularly broadcast a query to the input device fora lateral position, a longitudinal position, and a force magnitude ofeach discrete input area on the touch sensor surface, such as in theform of one or more current touch images at a rate of 60 Hz (or 60 Hz,or between 90 Hz and 120 Hz). The computer system can then process eachtouch image remotely from the input device, including identifyingdiscrete input areas, calculating a peak force per discrete input area,calculating a total force per discrete input area, labeling orcharacterizing a type of physical object contacting the touch sensorsurface and corresponding to each discrete input area, etc., such asdescribed below. Alternatively, the input device can locally process atouch image before uploading the touch image and/or data extracted fromthe touch image (e.g., types, force magnitudes, and locations ofdiscrete input areas) to the computer system.

1.3 Real Position and Orientation

Block S110 of the first method S100 recites, at a first time,determining a first position of a touch sensor within a real space, thetouch sensor including a touch sensor surface. Generally, in Block S110,the computer system functions to record a real position and orientationof the input device.

In one implementation, the computer system tracks and records theposition and orientation of the input device in six degrees of freedom,such as including: a lateral (e.g., X-axis) position; a vertical (e.g.,Z-axis) position; a depth (e.g., Y-axis) position; a pitch orientation(e.g., angular position about the Y-axis); a yaw orientation (e.g.,angular position about the Z-axis); and a roll orientation (e.g.,angular position about the Z-axis) of a reference point of the inputdevice relative to a static coordinate system (or static referencepoint) within the real space. For example, the computer system can trackthe position and orientation of a center of the input device, areference corner of the input device, or other reference point on theinput device relative to a position tracking base station, a virtualreality (VR) headset, an augmented reality (AR) headset, or othercomputer housing or sensor.

In one implementation, the computer system interfaces with athree-dimensional position sensor—such as a LIDAR sensor or a structuredlight three-dimensional sensor occupying a common space with the inputdevice—to scan the real space for the input device; by regularlysampling the three-dimensional position sensor during operation, thecomputer system can track the position and orientation of the inputdevice over time. For example, the input device can incorporate two ormore quick-response codes, infrared or photo emitters, and/or otheractive or passive fiducials arranged on surfaces of the input device. Acamera or optical sensor(s) arranged within the real space with theinput device within the field of view captures and outputs image framesto the computer system. In this example, the computer system implementscomputer vision techniques to identify the fiducials in each imageframe, correlates relative positions of the fiducials with anorientation of the input device and a distance of the input device fromthe camera, and determines lateral and vertical positions of the inputdevice within the real space relative to the camera based on positionsof the fiducials within an image frame. In another example, the inputdevice incorporates and selectively drives three perpendicular magneticcoils, and a base station (arranged within the real space) includesthree perpendicular magnetic field sensors that it samples to trackstrengths and orientations of magnetic fields output from the inputdevice; the base station then calculates the position and orientation ofthe input device—during the sampling period—relative to the base stationaccording to the strengths and orientations of these magnetic fields.However, the computer system can cooperate with the input device and/oranother peripheral device to calculate the position and orientation ofthe input device in each sampling (or scan) period.

Alternatively, the input device can track its own position within thereal space—such as at a refresh rate of 60 Hz—and uploads valuesrepresenting its position and orientation to the computer system foreach sampling period substantially in real-time. In one example, theinput device samples integrated inertial sensors (e.g., a three-axisaccelerometer, a three-axis gyroscope, a compass), implements deadreckoning techniques to estimate the position of the input device ateach sampling period based on outputs of the inertial sensors, andwirelessly broadcasts position and orientation values to the computersystem. In another example, the input device includes a set of opticaldetectors that detect incident light emitted from one or more basestations arranged within the real space, and a processor within theinput device correlates positions and intensities of light incident onthe sensors during a sampling period with positions and orientations ofthe input device within the real space relative to the base station(s);in this example, the input device then broadcasts the calculatedposition and orientation of the input device for the sampling period tothe computer system. For a sampling (or scan) period at the inputdevice, the input device can also record and timestamp outputs ofinertial sensors integrated into the input device, and the input device(or the computer system) can transform these outputs into a realorientation (e.g., pitch, yaw, roll) of the input device—such asrelative to gravity and compass north as determined from outputs of theinertial sensors—for the sampling period.

However, the computer system can interface with the input device and/orany other integrated or external device or sensor to track the positionand orientation of the input device in real space across consecutivesampling periods during operation of the input device and computersystem.

In one implementation shown in FIG. 1, the input device augments a touchimage—containing input location and force magnitude data for the touchsensor surface collected during a particular scan period (as describedabove)—with a timestamp corresponding to the particular scan periodbefore pushing the touch image to the computer system in Block S130, asdescribed below. During operation, the computer system: transforms realposition and orientation tracking data of a reference point on the inputdevice at the particular (e.g., current) sampling period into a realposition and orientation of the touch sensor surface; maps the touchsensor surface to a virtual surface in the virtual environment inreal-time in Block S120 according to the real position and orientationof the touch sensor surface for the sampling period; matches the touchimage received from the input device to the current sampling period intime based on a similar timestamps; merges the real orientation of thetouch sensor surface and the touch image recorded at similar times intoa force vector and an anchor for the force vector for the particularsampling period in Blocks S140 and S142; and then manipulates a portionof the virtual surface corresponding to the position and orientation ofthe real surface according to the force vector and the anchor point ofthe force vector substantially in real-time (e.g., substantiallyimmediately after the times recorded in the first and second timestamps)in Block S150.

1.4 Real Surface to Virtual Surface

Block S120 of the first method S100 recites, based on the first positionof the touch sensor within the real space at the first time, bounding avirtual surface of a virtual object within the virtual environmenttractable (e.g., controllable, manipulatable) through inputs across anarea of the touch sensor surface. (Block S120 can similarly recite,based on a position of the touch sensor surface in the real space,mapping the touch sensor surface to a virtual surface in a virtualenvironment.) Generally, in Block S120, the computer system functions toproject a real surface (i.e., the touch sensor surface in real space)onto a virtual surface of a virtual object within the virtualenvironment to define a region of the virtual object that a user mayvirtually grasp, virtually translate, virtually rotate, virtually scale,or otherwise virtually manipulate by moving the input device in realspace, by moving a finger, stylus, or other object across the touchsensor surface, and/or by modulating a force applied to the touch sensorsurface.

In one implementation, the computer system maps a reference point on theinput device (e.g., on the touch sensor surface) to a virtual point on avirtual surface (or within a virtual object) within the virtualenvironment based on the lateral, vertical, and depth positions of thetouch sensor surface within the real space. Based on the orientation ofthe touch sensor surface within the real space, the computer system thenbounds a region of the virtual surface (or a virtual slice of thevirtual object) anchored by the virtual point and corresponding to thesize and geometry of the touch sensor surface. The computer system canthus define a general bounded virtual surface within the virtual modelthat can be manipulated by the user through touch inputs on the touchsensor surface. The computer system can also locate an origin of theforce vector to a specific position within the bounded region of thevirtual surface in Block S142 based on the position of an input appliedto the touch sensor surface. Specifically, the computer system candefine a global (or “macro”) region of a virtual object controllablethrough inputs on the touch sensor surface of the input device based onthe position of the touch sensor surface in real space and then define aspecific location of the origin of the force vector within the virtualenvironment based on the position of an input on the touch sensorsurface, such as the centroid of the area of an input or the location ofthe peak force measured within the area of the input, in Block S142. Forexample, the computer system can map the origin of the force vector to avertex, triangle, or point in a 3D mesh representing the virtualenvironment. In another example, the computer system can map the originof the force vector to a particular pixel within a texture mapped to the3D mesh representing the virtual environment.

In one implementation, the computer system locates a virtual referenceboundary representing the touch sensor surface within the virtualenvironment and projects rays from the virtual reference boundary ontothe virtual surface, as shown in FIG. 1, in order to map the touchsensor surface to the virtual surface. For example, the touch sensorsurface can define a substantially planar rectilinear area (e.g., a 120millimeter by 200 millimeter planar area), and the computer system canlocate a rectangular, 3:5 aspect ratio, virtual referenceboundary—representing the touch sensor surface—within the virtualenvironment based on the position and the orientation of the touchsensor surface in real space, project rays from the corners of thevirtual reference boundary (and normal to the plane of the virtualreference boundary) onto a nearest virtual surface within the virtualenvironment to define select points on the virtual surface, and connectadjacent points on the virtual surface to define a manipulatable regionof the virtual surface. Thus, in Block S120, the computer system can:project a rectangular reference boundary corresponding to a rectangular,planar area of the touch sensor surface onto the virtual surface of athree-dimensional virtual object according to the position and theorientation of the touch sensor within the real space; and can enablecontrol of the region of the virtual surface bounded by this projectionof the rectangular reference boundary on the virtual object throughmanipulation of the input device and according to inputs on the touchsensor, thereby mapping the touch sensor surface to the virtual surface.

In a similar implementation, the computer system: locates a virtualreference plane representing the touch sensor surface within the virtualenvironment based on the position and orientation of the touch sensorsurface in real space; projects rays normally outward from the virtualsurface—and toward a virtual reference plane—within the virtualenvironment; and maps real points on the touch sensor surface to virtualpoints on the virtual surface according to origins of rays emanatingfrom the virtual surface and intersecting corresponding points on thevirtual reference plane. The computer system can therefore project aregion of the virtual surface onto a virtual referenceboundary—representing the touch sensor surface—within the virtualenvironment to map the virtual surface to the touch sensor surface.

In the foregoing implementations, the computer system can set staticvirtual dimensions for the virtual reference boundary. Alternatively,the computer system can scale the virtual dimensions of the virtualreference boundary (e.g., with length-to-width ratio static) based on aposition of the input device in the real space, such as based on adistance between a reference point on the input device and a referenceorigin or reference plane assigned to the real space. For example, as auser moves the input device forward (relative to the reference planeassigned to the real space) to zoom into the virtual model, the computersystem can scale down the virtual dimensions of the virtual referenceboundary representing the touch sensor surface in the virtualenvironment, thereby reducing the size of the bounded region of thevirtual surface mapped to the touch sensor surface and enabling finerpositional control of the virtual surface (or of a virtual toolcontacting or acting on the virtual surface) within the virtualenvironment through the touch sensor surface. Similarly, in thisexample, as the user moves the input device backward to zoom out fromthe virtual model, the computer system can increase the scale of thevirtual reference boundary representing the touch sensor surface in thevirtual environment, thereby increasing the size of the bounded regionof the virtual surface mapped to the touch sensor surface and enablingmore global manipulation of the virtual surface or of the virtual objectthrough the touch sensor surface. Therefore, in Block S120, the computersystem can: increase a scale of the (rectangular) reference boundary inresponse to movement of the touch sensor in a first direction relativeto a user within the real space; and decrease a scale of the rectangularreference boundary in response to movement of the touch sensor in asecond direction opposite the first direction within the real space.

When generating a force vector for a sampling period in Block S140, thecomputer system can additionally or alternatively scale a magnitude of aforce recorded in a touch image according to the position of the touchsensor surface within the real space. For example, given substantiallyconstant application of a force over an input area on the touch sensorsurface over a sequence of sampling periods, the computer system canincrease the force magnitude of force vectors generated over thesequence of sampling periods as the user pushes the input device forwardover the same period of time; and the computer system can reduce theforce magnitude of successive force vectors generated over the sequenceof sampling periods as the user pulls the input device back over thesame period of time.

The computer system can execute the foregoing methods and techniquesregularly throughout operation, such as once per sampling period at theinput device, to recalculate a virtual surface on the virtual object inthe virtual environment that is mapped to and manipulated via inputs onthe touch sensor surface in response to movement of the input devicewithin the real space (and thus relative to the virtual environment).Alternatively, the computer system can statically lock the (physical)input object to the virtual object, move the virtual object within thevirtual environment according to movements of the input device in thereal space, and preserve the virtual surface of the virtual objectmapped to the touch sensor surface over time. For example, for thevirtual object that represents a virtual tool, such as a pair of virtualpliers, the computer system can move the virtual pliers within thevirtual environment according to movement of the input device in thereal space, map the touch sensor surface (or a portion of the touchsensor surface) to handles of the virtual pliers, open and close thevirtual pliers according to the proximity of two inputs on the touchsensor surface, and close the pliers according to force magnitudes ofthe two inputs on the touch sensor surface.

In variations of the input device that include a curvilinear (e.g.,semi-cylindrical or semi-spherical) touch sensor surface, as describedbelow, the computer system can implement similar methods and techniquesto map (and scale) a curvilinear reference boundary—corresponding to thegeometry of the curvilinear touch sensor surface—to a planar orcurvilinear virtual surface of a virtual object within the virtualenvironment. For example, for the input device that defines acylindrical touch sensor surface, the computer system can virtually“unwrap” the cylindrical touch sensor surface into a planar referenceboundary and map this planar reference boundary to a virtual surface ofa virtual object in the virtual environment, as described above. Inanother example, for the input device that defines a cylindrical touchsensor surface, the system can axially align a cylindrical referenceboundary to an axis of a three-dimensional virtual object—of a geometryapproximating a cylinder—in the virtual environment and then project thecylindrical virtual reference boundary onto an adjacent region of thevirtual object to define a region of the virtual object in the virtualenvironment manipulatable via the touch sensor surface in the realspace.

1.5 Visual Feedback

In one implementation, the computer system renders a virtual markerrepresenting the size, shape, position, and/or orientation of the touchsensor surface within the virtual environment. For example, the computersystem can render a virtual solid or dashed white box corresponding tothe perimeter of the touch sensor surface within the virtualenvironment, such as floating within the virtual environment orprojected onto the region of the virtual surface of the virtual objectmapped to the touch sensor surface. In this example, the computer systemcan update the position of virtual box within the virtual environmentfor each consecutive sampling period based on the real position andorientation of the touch sensor surface in real space. The computersystem can also scale the virtual box within the virtual environment asthe user moves the input device forward and backward (or in any otherdirection) within the real space.

Similarly, the computer system can render a virtual tool (e.g., virtualpliers, a virtual gun) or a virtual hand in the virtual environmentrepresenting the input device in the real space. The computer system canalso move (e.g., translate, rotate) the virtual tool or virtual hand inthe virtual environment according to movement of the input device in thereal space. Furthermore, the computer system can modify or manipulatesurfaces or volumes of the virtual tool or virtual hand based onpositions and force magnitudes of inputs on the touch sensor surface.For example, the computer system can: move a virtual trigger of avirtual gun a distance corresponding to a force magnitude of an input onthe touch sensor surface at a location corresponding to the virtualtrigger; close a pair of virtual pliers a virtual distance and with avirtual clamping force corresponding to the positions and forcemagnitudes, respectively, of two inputs on the touch sensor surface, asdescribed above; or move virtual fingers of a virtual hand in adductionor abduction according to the positions of inputs on the touch sensorsurface and move these virtual fingers in flexion and extensionaccording to force magnitudes of these inputs on the touch sensorsurface, as described below.

1.6 Force Vector

Block S140 of the first method S100 recites generating a first forcevector including a magnitude related to a force magnitude of a firstinput on the touch sensor surface and a direction related to anorientation of the touch sensor within the real space at the first time.(Block S140 can similarly recite, based on an orientation of the touchsensor surface in the real space, assigning a three-dimensionaldirection to the force magnitude to construct a force vector.) The firstmethod S100 can also include Block S142, which recites locating anorigin of the first force vector within the virtual environment based ona first location of the first input on the touch sensor surface and thefirst position of the touch sensor within the real space. Generally, inBlocks S140 and S142, the computer system combines a magnitude of aforce applied to the touch sensor surface with an orientation of thetouch sensor surface within the real space (within a sampling or scanperiod) and the location of the applied force on the touch sensorsurface to generate a virtual force vector representing a magnitude,direction, and origin of a virtual force to be applied to a virtualsurface (or to the virtual object generally) within the virtualenvironment to manipulate the virtual object in Block S150.

As described above, the input device can output a touch imagerepresenting a magnitude of applied forces detected by each senseelement in the touch sensor (and interpolated force magnitudes betweensense elements in the touch sensor) during a corresponding scan period,as shown in FIG. 1; and the computer system can determine the realposition and orientation of the input device—such as relative to anorigin or relative to a coordinate system assigned to the real spaceoccupied by the input device—at the same or similar time duringoperation of the input device and computer system. To generate a forcevector for a scan period, the computer system can: identify a discreteinput area represented in the touch image (e.g., a contagious area ofelevated applied force values bounded by baseline force values in thetouch image); store the (X,Y) location of the centroid of the discreteinput area or the (X,Y) location of a point of peak force magnitude inthe discrete input area as the location of a corresponding input on thetouch sensor surface; calculate a direction of a real force vectornormal to and facing into the touch sensor surface, such as relative toan origin of a coordinate system assigned to the real space; record areal force magnitude of the discrete input area, such as a peak forcemagnitude, a total force magnitude, or an average force magnitude withinthe discrete input area. The computer system can then: pair the realdirection with a real magnitude of the applied force represented in thetouch image to generate a “real force vector”; and translate the realforce vector into a “virtual” force vector by applying atransformation—that maps the coordinate system assigned to the realspace to a coordinate system in the virtual environment—to the realforce vector in Block S140.

The computer system can also anchor an origin of the virtual forcevector within the virtual environment by mapping the (X,Y) position ofthe discrete input area on the touch sensor surface to a point on thevirtual surface in Block S142. For example, the computer system canproject a normal ray from a point on a virtual referenceboundary—representing the perimeter of the touch sensor surface, asdescribed above, and corresponding to the position of the force on thetouch sensor surface—within the virtual environment onto the virtualsurface and anchor the virtual force vector at a point on the virtualsurface intersected by the normal array. Alternatively, the computersystem can anchor the origin of the virtual force vector within thevirtual environment by: determining a real position of the discreteinput area in real space based on the (X,Y) location of the input on thetouch sensor surface and the position and orientation of the inputdevice in real space; and then applies the transformation describedabove to the real position of the discrete input area to locate theorigin of the virtual force vector in the virtual environment.

The computer system can implement the foregoing methods and techniquesfor each input on the touch sensor surface—corresponding to a total orpeak force magnitude above a baseline force—detected by the touch sensorand recorded in a touch image for each consecutive scan period duringoperation of the input device and computer system. The computer systemcan thus generate one force vector for each discrete input on the touchsensor surface represented in each touch image output by the inputdevice, as shown in FIG. 1.

1.7 Motion Vector

The computer system can also generate a motion vector representing achange in position and/or orientation of the input device between two(or more) consecutive sampling periods. The computer system can thenmove the virtual object globally according to this motion vector andmove, modify, deform, or otherwise manipulate the virtual object locally(e.g., at the virtual surface mapped to the touch sensor surface) basedon the position and force magnitude of one or more inputs on the touchsensor surface.

In one example in which the virtual object includes a virtual baseball,the computer system can virtually grip the virtual baseball in a virtualhand—linked to the input device—based on a magnitude of a forcemagnitude of one or more inputs applied to the touch sensor surface, andthe computer system can translate and rotate the virtual baseball withinthe virtual environment according to changes in the position of theinput device in the real space. As a user moves the input device rapidlyforward (e.g., over his right shoulder and forward) in the real spaceand releases one or more inputs on the touch sensor surface (e.g.,integrated into a cylindrical or spherical input device with a wriststrap that retains the input device near the user's palm, as describedbelow), the computer system can generate a motion vector for the virtualbaseball corresponding to the real trajectory of the input device inreal space and virtually “release” the virtual baseball in the virtualenvironment according to the motion vector upon release of the input(s)on the touch sensor surface. Furthermore, in this example, the computersystem can: map the touch sensor surface to a virtual surface of thevirtual baseball; track the location and force magnitudes of inputs onthe touch sensor surface to calculate and locate a force vector for eachinput on the touch sensor surface; and release the virtual baseball withvirtual spin based on the magnitude of a particular force vectorcorresponding to a last input on the touch sensor surface, the locationof the particular force vector relative to the virtual center of mass ofthe virtual baseball, and the angle of the particular force vector tothe virtual surface of the virtual baseball as all other inputs arewithdrawn from the touch sensor surface to release to the virtualbaseball in the virtual environment.

1.8 Calibration

One variation of the first method S100 further includes: at an initialtime preceding the first time, serving a prompt to a user to apply aforce of subjective qualitative value to the touch sensor surface;recording a magnitude of an initial force applied to the touch sensorsurface responsive to the prompt; and scaling the magnitude of the firstforce vector based on the magnitude of the initial force. Generally, inthis variation, the computer system can interface with the input deviceto calibrate a scale factor for transforming a force magnitude of aninput on the touch sensor surface into a magnitude of the virtual forcevector to a user (e.g., to the user's size or strength).

In one example, the computer system can issue a prompt to a user atstartup (e.g., when entering or about to enter a virtual environmentthrough a virtual reality headset) to sequentially press “lightly,” topress with “moderate force,” and/or to press “firmly” onto the touchsensor surface with a finger (or with a stylus or other object) and thencollect touch images from the input device as the user enters one ormore such inputs on the touch sensor surface. Based on the peak or totalforce magnitude of each input, the computer system can: adjust forcemagnitude thresholds that then must be exceeded by (or exceed) an inputon the touch sensor to trigger an action within the virtual environment;and/or recalculate a scale factor for transforming a force magnitude ofan input on the touch sensor surface into a magnitude of the virtualforce vector. In a similar example, the computer system can prompt auser—such as through a heads-up display in a virtual reality oraugmented reality headset worn by the user—to enter a force of magnitude“5” on a scale of “1” to “10” and then adjust the scale factor fortransforming an input on the touch sensor surface into a force vectoraccording to the peak or total force at an input area on the touchsensor surface represented in a subsequent touch image received from theinput device.

The computer system can thus execute a calibration routine duringoperation in order to enable users of different sizes andstrengths—interfacing with the same input device and computer systemover time—to apply personalized ranges of forces on the touch sensorsurface to enter similar control commands into the virtual environment.However, the computer system can implement any other method or techniqueto calibrate a force magnitude scale factor for a user interfacing withthe input device.

Furthermore, the touch sensor surface on the input device can define acontinuous surface without discrete input positions, and the computersystem can map input controls to discrete input areas represented in atouch image received from the input device, thereby enabling users withdifferent hand sizes (e.g., both a five-year-old chair and an adultmale) to manipulate similar virtual objects in similar ways withinsimilar virtual environments through an input device and touch sensorsurface of one size and configuration. For example, upon selecting theinput device, a user can place her fingers on the touch sensor surfacein a position that she finds comfortable, the input device can output atouch image that represents the position of the user's fingers on thetouch sensor surface, and the computer system can calibrate controlfunctions for the virtual environment to these discrete input areasrepresented in the touch image, such as by mapping control functions tothe positions of the user's fingers represented in the touchimage—rather than require the user to place her fingers over predefininginput regions on the input device.

1.9 Modifying the Virtual Surface

Block S150 of the first method S100 recites modifying the virtualsurface according to the vector within the virtual environment.Generally, in Block S150, the computer system manipulates a virtualsurface (or a virtual object) within the virtual environment accordingto the force magnitude and direction of a virtual force vector generatedin Block S140.

In one implementation and as shown in FIG. 1, the computer system movesand/or deforms a virtual surface (or a virtual object) within thevirtual model according to a virtual force vector and a physics modeldefining mechanical creep, elastic deformation, plastic deformation,and/or inertial dynamics, etc. of the virtual surface and/or the virtualobject. In particular, the computer system can implement the physicalmodel to: calculate deformation of a virtual surface mapped to the touchsensor surface; calculate changes in the position and orientation of avirtual object; and calculate inertial effects on the virtual objectresulting from a virtual force—of magnitude, direction, and position(e.g., origin) defined in the virtual force vector—applied to thevirtual surface over the duration of one scan period. The computersystem can update the virtual surface (or the virtual object) renderedwithin the virtual environment accordingly in Block S160 describedbelow, such as by generating a new two-dimensional or three-dimensionaldigital frame representing the virtual environment given a currentviewing position of the user and then uploading the new frame to adigital display, VR headset, or AR headset for presentation to the usersubstantially in real-time.

For consecutive scan periods in which the input device generates anduploads touch images representing one or more input forces to thecomputer system, the computer system can also smooth virtual objectmanipulations between consecutive scan periods and/or remove signalnoise across the input device and computer system by averaging positionsand magnitudes of forces on the touch sensor surface over a sequence ofconsecutive scan periods. For example, the computer system can track oneinput on the touch sensor surface in five touch images corresponding tothe last five scan periods at the touch image and average the positionsand force magnitudes of this one input over the last five touch imageswith the current scan period given greatest weight to calculate a forcevector for the current scan period. The computer system can also averagepositions and orientations of the input device over a similar sequenceof sampling periods, such as the last five sampling periods up to thecurrent sampling period with the current sampling period given greatestweight to calculate a motion vector, as described above, for the currentsampling period.

In one example implementation in which the input device includes asingle, substantially flat (e.g., planar) pressure-sensitive touchsensor and one touch sensor surface, such as shown in FIGS. 1 and 6, thecomputer system transforms a force applied to the touch sensor surfaceinto a local inward deformation of a virtual surface of a virtual objectin the virtual environment—corresponding to the touch sensor surface inreal space—according to a force vector generated according to theposition and force magnitude of an input on the touch sensor surface andthe position and orientation of the touch sensor surface in real spacein Block S150. In this example implementation, the computer system candeform a local region of the virtual surface that intersects the virtualforce vector; the perimeter of the deformed local region can define asize and geometry related to the size and geometry of the contact areaof the input on the touch sensor surface; deformation within the localregion can correspond to the force profile across the correspondinginput on the touch sensor surface; and the location and magnitude ofmaximum deformation of the local region of the virtual surface alsocorresponding to the location and magnitude of the peak force within thecorresponding input area. The computer system can also “stretch” aperipheral region of the surface around the deformed region as the usercontinues to apply a force to the touch sensor surface over a series ofconsecutive scan periods, such as based on a modulus of elasticitydefined in a physical model assigned to the virtual object. In thisexample implementation, to pull the deformed region of the virtualsurface back to its previous contour, the user can flip the input device180° about its X- or Y-axis (i.e., turn the input device around to facethe back side of the input device) and apply an input at a similarlocation and of a similar force magnitude on the touch sensor surface;the computer system can thus detect this new position of the inputdevice, identify an input on the touch sensor surface at a locationcorresponding to the deformed region of the virtual surface, and deformthe local region of the virtual surface outwardly according to the forcemagnitude and the position of the input on the touch sensor surface andthe position and orientation of the touch sensor surface in real space.

In a similar example implementation in which the input device includes afirst touch sensor surface on one side of the input device and a secondtouch sensor surface on an opposite side of the input device, thecomputer system can deform (or “push”) a region of a virtual surfaceinwardly when a force is applied to the first touch sensor surface andcan deform (or “pull”) the region of the virtual surface outwardly whena force is applied to an opposing location on the second touch sensorsurface.

In one example application, the computer system interfaces with theinput device—including a first touch sensor on one side of the inputdevice and a second touch sensor on a second side of the input deviceopposite the first touch sensor—to control a virtual clay wheel-throwingenvironment. In this example application, the computer system: generatesdigital (two- or three-dimensional) frames of a virtual environmentincluding a potter's wheel, a virtual mass of clay, and virtual hands;and translates a force profile across a contact area of an input objecton the touch sensor surface into a virtual finger (or fingers, palm,hand) that works the virtual clay mass spinning on the virtual potter'swheel. For example, as a user depresses a thumb on the first touchsensor, the computer system can update frames of the virtual environmentto show a virtual thumb (or first virtual forming tool) pushing thevirtual clay mass inward toward the center of the virtual potter's wheelaccording to the position and force magnitude of the thumb input and theposition and orientation of the first touch sensor in the real space.Similarly, as the user depresses an index finger on the second touchsensor—on the opposite side of the input device—the computer system canupdate frames of the virtual environment to show a virtual index finger(or second virtual forming tool) contacting the virtual mass of clay,forming a virtual interior wall in the mass of virtual clay spinning onthe virtual potter's wheel, and pulling the virtual clay wall outwardfrom the center of the virtual potter's wheel. In this example, as theuser depresses a thumb onto the first touch sensor and an index fingeron the second touch sensor (i.e., as the user pinches the input devicebetween the first and second touch sensors), the computer system cangenerate subsequent digital frames of the virtual environment thatdepict virtual fingers compressing the virtual wall on the potter'swheel according to a measured compressive force applied by the useracross the first and second touch sensor surfaces. Furthermore, as theuser continues to pinch the input device and to move the input devicethrough an arc in a vertical plane in the real space, the computersystem can generate images of the virtual environment depicting virtualfingers compressing the virtual clay wall and drawing the virtual claywall in an upward direction—corresponding to the arc of the input devicein real space—to form a virtual clay vessel spinning on virtual potter'swheel. In particular, as the user articulates the input device upward inreal space and pinches the input device between the first and secondtouch sensors with an increasing compressive force, the computer systemcan translate this real compressive force on the input device and themotion of the input device into transformation of the virtual clay massinto a virtual clay wall that tapers upwardly toward its top edge toform a virtual clay vessel of height and form related to the distanceand trajectory of the arc traversed by the input device. In this exampleapplication, once the virtual clay vessel is complete, the computersystem can translate the virtual clay vessel into a manufacturing fileand upload the manufacturing file to an additive or subtractivemanufacturing system or service for recreation in physical material.

1.10 Moving the Virtual Surface

Block S150 of the first method S100 can similarly recite manipulatingthe virtual surface of the virtual object within the virtual environmentaccording to the first force vector. Generally, in Block S150, thecomputer system can translate (or rotate) a virtual object within thevirtual environment based on a position, direction, and force magnitudeof the force vector generated in Block S140.

In one implementation, the computer system projects the force vectoronto a point on the virtual surface of the virtual object and then movesthe virtual object in three dimensions within the virtual environmentaccording to the magnitude of the first force vector acting on the pointon the virtual surface in the direction of the first force vector, suchas in the example described above in which the virtual object includes avirtual baseball. The computer system can thus move a virtual object inmultiple degrees of freedom within the virtual environment based on aphysic or motion model associated with the virtual object. In thisimplementation, the computer system can also implement methods andtechniques described above to temporarily deform the virtual surface ofthe virtual object—proximal the point on the virtual surface intersectedby the force vector—according to the magnitude and direction of theforce vector.

In another implementation, Block S150 includes: tracking a position of atouch sensor within a real space; locating a virtual object globally inthree dimensions within the virtual environment based on the position ofthe touch sensor in the real space; translating the virtual objectlocally along a first axis and a second axis within the virtualenvironment according to a trajectory of an input moving across thetouch sensor surface, wherein the first axis is related (e.g., mapped)to a width (e.g., an X-axis) of the touch sensor surface, and whereinthe second axis is perpendicular to the first axis and is related to aheight (e.g., a Y-axis) of the touch sensor surface; and translating thevirtual object locally along a third axis within the virtual environmentaccording to a force magnitude of the input on the touch sensor surface,the third axis perpendicular to the first axis and the second axis andrelated to a depth of the touch sensor surface.

In the foregoing implementation, the computer system can thus modulate aZ-position (or a “depth”) of the virtual object within the virtualenvironment—relative to a virtual reference plane corresponding to thetouch sensor surface in real space—based on a force magnitude of aninput on the touch sensor surface. For example, the computer system canmove the virtual object by a first distance in a first direction (e.g.,forward away from the user)—relative to the virtual reference plane—inthe virtual environment in response to the force magnitude of an inputon the touch sensor surface exceeding a threshold force magnitude; inthis example, the computer system can calculate the first distance as afunction of a difference between the force magnitude of the input (orthe magnitude of the force vector) and the threshold force magnitude.Similarly, the computer system can move the virtual object by a seconddistance in a second direction opposite the first direction (e.g.,backward toward the user)—relative to the virtual reference plane—in thevirtual environment in response to the force magnitude of the input (orthe magnitude of the force vector) falling below the threshold forcemagnitude; in this example, the computer system can calculate the seconddistance as a function of the difference between the force magnitude andthe threshold force magnitude. Therefore, in this implementation, thecomputer system can position a virtual object globally within thevirtual environment based on the position and orientation of the inputdevice in real space. However, the computer system can move the virtualobject in a Z-axis (e.g., normal to a virtual reference planecorresponding to the touch sensor surface) over a smaller distance rangebased on the magnitude of a force applied to the touch sensor surface,thereby enabling the user to move the virtual object globally within thevirtual environment by moving the input device in real space but alsoenabling the user to offset the virtual object in the virtualenvironment from the input device in real space along a Z-axis of theinput device by modulating a force magnitude of an input applied to thetouch sensor surface. Similarly, the computer system can move thevirtual object locally over a relatively small distance range indirections parallel to the X- and Y-axes of the virtual reference planeas the input is moved in X- and Y-directions across the touch sensorsurface, thereby enabling the user to also offset the virtual object inthe virtual environment from the input device in real space along bothX- and Y-axes of the input device by moving a finger or stylus acrossthe touch sensor surface. In particular, the computer system canposition and orient a virtual object globally within the virtualenvironment based on the real position of the input device in the realspace and shift the location of the virtual object in the virtualenvironment locally relative to the X-, Y-, and Z-axes of the inputdevice based on changes in the position of an input on the touch sensorsurface and the force magnitude of such an input on the touch sensorsurface.

In a similar implementation, the computer system can: translate androtate the virtual object within the virtual environment according tochanges in the position and orientation of the touch sensor within thereal space; and locally deform a surface of the virtual object at alocation corresponding to the position of an input on the touch sensorsurface and to a degree related to the force magnitude of the input onthe touch sensor surface. In one example, in which the virtual objectincludes a virtual water balloon, the computer system can: move thevirtual water balloon in six degrees of freedom within the virtualenvironment based on changes in the location and orientation of theinput device within the real space; and locally deform (e.g., “squish”)the virtual water balloon between two virtual fingers located within thevirtual environment according to the location of two discrete inputs onthe touch sensor surface and to a degree related to force magnitudes ofthe two discrete inputs on the touch sensor surface.

1.11 Grasping the Virtual Object

In another implementation in which the input device includes a singletouch sensor defining a single substantially flat (e.g., planar) touchsensor surface, the computer system can: translate one or more inputs onthe touch sensor surface into a “grasping” or “pinching” gesture ofmagnitude corresponding to the force magnitude of the input(s); link theposition of the virtual object within the virtual environment to theinput device in the real space based on the grasping or pinchinggesture; and manipulate the virtual object within the virtualenvironment based on the force magnitude of the input(s) on the touchsensor surface.

For example, in this implementation, the touch sensor can: detect afirst input at a first location on the touch sensor surface at a firsttime; detect a second input at a second location offset from the firstinput on the touch sensor surface at the first time; and upload theseinput data to the computer system in the form of a touch image for onescan period. The computer system can then: label the first input and thesecond input as a “pinch” input in response to the second input movingrelatively toward the first input; locate a virtual tool within thevirtual environment proximal the virtual object based on the position ofthe touch sensor at the first time; position two virtual elements (e.g.,two virtual fingers, tips of virtual pliers) contiguous with (e.g.,extending from) the virtual tool onto opposing sides of the virtualobject based on the first location of the first input, the secondlocation of the second input on the touch sensor surface, and thedirection of the first force vector in Block S150. In particular, thesystem can locate the virtual tool globally near the virtual objectbased on the position of the input device in real space and manipulatethe local positions of two (or more) virtual elements on the virtualtool based on the locations of corresponding inputs on the touch sensorsurface, including moving the virtual elements together as correspondinginputs on the touch sensor move near one another, as determined bytracking discrete inputs on the touch sensor surface over a sequence ofscan periods through touch images received from the touch sensorsurface. Furthermore, once the virtual elements contact the virtualobject, the computer system can couple the virtual elements and thevirtual tool to the virtual object with a firmness related to the forcemagnitude of one or both inputs on the touch sensor surface (e.g.,according to the magnitude of first and/or second force vectorsgenerated for the first and second inputs on the touch sensor surface inBlock S140).

In the foregoing implementation, the computer system can then move thevirtual object with the virtual tool within the virtual environmentaccording to a change in the position of the touch sensor within thereal space while the force magnitude of the first input and/or the forcemagnitude of the second input remain above a threshold force magnitude.The computer system can also “slip” the virtual object relative to thevirtual tool as the force magnitude of the first input and/or the forcemagnitude of the second input decrease relative to a speed with whichthe input device is moved throughout the real space. Finally, thecomputer system can decouple the virtual tool from the virtual objectwithin the virtual environment in response to the force magnitude of thefirst input and/or the second input falling below the threshold forcemagnitude. For example, the computer system can implement a thresholdforce magnitude for grasping the virtual object that is related to avirtual weight of the virtual object such that a heavier virtual objectrequires inputs of greater force magnitude on the touch sensor surfaceto grasp the heavier virtual object than a light virtual object. Thecomputer system can thus enable a user to grasp, pinch, or otherwiseselect, manipulate, and then release a virtual object within the virtualenvironment by moving the input device within the real space, movinginputs across the touch sensor surface, and modulating the forcemagnitude of the inputs on the touch sensor surface, and the computersystem can control a degree to which the virtual tool is coupled to thevirtual object based on the magnitude of a force applied to the touchsensor surface (and represented in a touch image).

In the foregoing implementation, the virtual tool can define a virtualhand, and the virtual elements can define discrete virtual fingers. Thecomputer system can: move the virtual hand within the virtualenvironment based on changes in the position and orientation of theinput device within the real space; move a virtual finger on the virtualhand in adduction and abduction based on a change in the (X,Y) positionof an input on the touch sensor surface corresponding to the finger;move the finger to greater degrees of flexion as the force magnitude ofthe input increases; and move the finger to greater degrees of extensionas the force magnitude of the input decreases. The computer system canthus curl virtual fingers around a virtual object to virtually grasp thevirtual object in the virtual hand as force magnitudes of inputscorresponding to virtual fingers on the hand increase; and the computersystem can release the virtual object from the virtual hand as inputs onthe touch sensor surface corresponding to these virtual fingers arewithdrawn from the touch sensor surface. In particular, the computersystem can: translate and rotate a virtual hand within the virtualenvironment according to changes in the position and orientation of thetouch sensor within the real space; locally move a virtual finger on thevirtual hand along a Z-axis of the finger (e.g., flexion and extension)by a distance corresponding to the force magnitude of a correspondinginput on the touch sensor surface; and translate the virtual finger onthe virtual hand in the X-Y plane of the finger (e.g., adduction,abduction) according to the position of the corresponding input on thetouch sensor surface.

The computer system can implement similar methods and techniques tolabel two (or more) inputs occurring simultaneously on onethree-dimensional (e.g., semi-cylindrical or semi-spherical) touchsensor surface or occurring simultaneously on two discrete touch sensorsspanning two or more unique planes on the input device as a graspinginput. For example, the computer system can articulate virtual fingerson a virtual hand according to positions and force magnitudes of inputson one or more touch sensors on a cylinder input device, as describedbelow.

For example, the computer system can implement the foregoing methods andtechniques to grasp a virtual slingshot in the virtual fingers of avirtual left hand based on the force magnitudes of corresponding inputson a cylindrical left input device held in a user's left hand (e.g.,based on a “grasp” gesture) and to then move the virtual slingshot inthe virtual environment based on subsequent positions of the left inputdevice in the real space. In this example, the computer system can movea virtual right hand near a virtual sling on a virtual band coupled tothe virtual slingshot held in the virtual left hand based on theposition of a cylindrical right input device in real space. Based on thelocations and force magnitudes of inputs on one or more touch sensors ofthe right input device (e.g., based on a “pinch” gesture), the computersystem can articulate virtual fingers on the virtual right hand to graspthe virtual sling of the virtual slingshot, as shown in FIG. 3. As theuser draws the right input device away from the left input device, thecomputer system can stretch the virtual band of the virtual slingshotbut also increase a threshold force magnitude of the grasp gesture and athreshold force magnitude of the pinch gesture at the right input devicerequired to retain the virtual slingshot in the virtual right and lefthands proportional to the distance between the left and right inputdevices in real space. In particular, if the user does not apply inputsof increasingly greater force magnitudes to the left and right inputdevices while separating the left and right input devices in physicalspace to wind the virtual slingshot, the computer system can “slip” theslingshot from the virtual left hand or slip the sling from the virtualright hand if corresponding force magnitudes of inputs on the left andright input devices do not remain above the increasing force magnitudethresholds assigned to the slingshot and to the sling by the computersystem. In particular, if the user does not grasp the right input devicewith an increasing force that remains above the increasing thresholdforce magnitude of the pinch gesture while drawing the right inputdevice away from the left input device, the computer system can slip thevirtual sling from the virtual right hand and fire a virtual projectilefrom the virtual slingshot at a speed and trajectory corresponding tothe relative position of the right input device to the left input deviceat the slippage time. For example, the computer system can implement afriction model (e.g., a static or kinetic friction model) to definepersistence or rigidity of coupling between the virtual right hand andthe virtual sling and to define slippage between the virtual right handand the virtual sling according to the force magnitude of the pinchgesture at the right input device.

However, if the user maintains a suitable force (or pressure) on theleft and right inputs devices while separating the left and right inputdevices to stretch the virtual band of the virtual slingshot, thecomputer system can preserve a connection between the virtual left handand the slingshot grip and between the virtual right hand and thevirtual sling until the user intentionally releases the pinch gesturefrom the right input device, at which point the computer system cansimilarly fire a virtual projectile from the virtual slingshot at aspeed and trajectory corresponding to the relative position of the rightinput device to the left input device within the real space at the timethe right input device was released by the user, as shown in FIG. 3. Thecomputer system can implement similar methods and techniques tomanipulate a virtual bow and arrow within the virtual environment basedon inputs into left and right input devices held by a user and relativepositions of these input devices.

In another example, the system can implement similar methods andtechniques within a virtual rock climbing environment to grasp a virtualhold with a virtual hand as a user manipulates an input device into aposition and orientation in real space corresponding to the virtual holdand then squeezes one or more fingers onto a touch sensor surface of theinput device (e.g., a cylindrical touch sensor surface, as describedbelow) to close virtual fingers on the virtual hand around the virtualhold. The computer system can thus: couple the virtual hand to thevirtual hold according to a force magnitude with which the userdepresses or squeezes the input device; lift a virtual body up a virtualclimbing wall as the user moves the input device downward in the realspace; and release the virtual hand from the virtual hold when the userreleases or withdraws one or more inputs on the touch sensor surface.However, if the force magnitude of inputs on the input device dropsbelow a threshold force magnitude corresponding to a virtual weight ofthe virtual body or if the force magnitude of inputs on the input devicedrops or does not increase to a threshold force magnitude before asecond virtual hand in the virtual environment is released from anothervirtual hold, the computer system can “slip” the virtual hold from thevirtual hand and drop the virtual body down the virtual climbing wall.

In yet another example, the system can implement similar methods andtechniques within a virtual water fight environment to grasp a virtualwater balloon with a virtual hand as a user manipulates an input deviceinto a position and orientation in real space corresponding to thevirtual water balloon within a basket of virtual water balloons and thensqueezes one or more fingers onto a touch sensor surface of the inputdevice (e.g., a cylindrical touch sensor surface, as described below) toclose virtual fingers on the virtual hand around the virtual waterballoon. However, if the user squeezes the touch sensor surface of theinput device with a force magnitude greater than a threshold burst forcemagnitude, the computer system can virtually burst the virtual waterballoon. Similarly, if the user squeezes the touch sensor surface of theinput device with a force magnitude less than a threshold holding forcemagnitude, the computer system can virtually “slip” the virtual waterballoon from the virtual hand and onto the ground (and virtually burstthe virtual water balloon).

In another example application, the computer system interfaces with theinput device to control a virtual toy brick-building environment. Inthis example application, the computer system generates digital framesof the virtual environment including a virtual brick model and virtualbricks in a virtual brick pile. For example, while holding an inputdevice with the touch sensor surface facing upward, a user can: move theinput device over a first position in real space corresponding to thevirtual brick pile in the virtual environment; pinch the touch sensorsurface of the input device to select a virtual brick from the virtualbrick pile; move the input device (with fingers still in contact withthe touch sensor surface) to a second position in real spacecorresponding to the virtual brick model; move fingers to a specificlocation on the touch sensor surface corresponding to a desired installposition for the virtual brick onto the virtual brick model; rotatefingers to set an orientation of the selected virtual brick in a virtualplane corresponding to the orientation of the touch sensor surface inreal space; and push downward on the touch sensor surface (or move theinput device downward) to install the virtual brick at the selectedposition on the virtual brick model. The computer system can thus trackforce values and positions of contact areas on the touch sensor surface,track the position and orientation of the input device within the realspace, and update a virtual toy brick-building environment accordinglysubstantially in real-time

1.12 Manipulating a Virtual Tool

In a another example application, the computer system interfaces withthe input device to control a virtual sculpting environment, as shown inFIG. 2. In this example application, the computer system generatesdigital frames of a virtual environment including a virtual mass of clayand virtual clay working tools (e.g., hands, fingers, scrapers, othermodeling tools, etc.). In particular, in this example application, thecomputer system translates a force profile across a contact area of aninput object on the touch sensor surface into a virtual clay modelingtool profile that deforms, cuts, and/or drags virtual clay within thevirtual environment. For example, the computer system can translate aninput passing across a surface of the touch sensor surface along anarc—and varying from a low peak pressure at a first end of the arc to ahigher peak pressure near the center of the arc and back to a low peakpressure at a second end of the arc—into a concave relief on a surfaceof the virtual clay mass by mapping the arc to a virtual surface on thevirtual clay mass and then removing (or compressing) virtual claymaterial within the virtual model along the (virtual) arc according tothe input force profile across the touch sensor surface. In thisexample, if the user contacts the touch sensor surface with a finger,the input device can record a force profile across the finger contactarea, including a greatest force near the center of the contact area anddecreasing force from the center of the contact area toward theperimeter of the contact area; the computer system can thus remove (orcompress) virtual clay material along and perpendicular to the arc basedon the force profile of the contact area through a series of consecutivesampling periods. In another example, the computer system transforms aninput on a rear (e.g., the second) touch sensor surface of the inputdevice into a stretching or pulling action on a corresponding region ofthe virtual clay mass, such as by translating a force magnitude on thetouch sensor surface into tension on a corresponding point or area ofthe virtual clay mass along a direction corresponding to the orientationof the input device at the same or similar time (e.g., for anoverlapping sampling period). In this example application, the computersystem can also push, pull, drag, twist, or otherwise manipulate thevirtual clay mass within the virtual environment according to inputforce magnitudes on the input device and orientations of the inputdevice in real space.

In another example application, the computer system interfaces with theinput device to control a virtual painting environment. In this exampleapplication, the computer system generates digital frames of a virtualenvironment including a virtual canvas, virtual paint, virtualpaintbrush, and virtual palette and translates a force profile across acontact area of a real paintbrush on the touch sensor surface intocontrolled collection of paint onto the virtual paintbrush andcontrolled deposition of paint into the virtual canvas and onto thevirtual palette. For example, a user can move the input device to a realposition corresponding to the virtual palette within the virtualenvironment, dab a real paintbrush onto a position on the input devicecorresponding to a virtual daub of paint on the virtual palette tovirtually collect a small amount of virtual paint on the tip of thevirtual paintbrush; the computer system can thus map the real positionof the touch sensor surface in real space to the virtual palette, mapthe contact area of the real paintbrush on the touch sensor surface tothe daub of paint on the virtual palette, and manipulate the virtualpaintbrush within the virtual environment according to the forceprofile, contact area, and trajectory of the real paintbrush across thetouch sensor surface. In this example, the user can also: twist the realpaintbrush over a first area of the input device corresponding to avirtual daub of a first paint color on the virtual palette to virtuallycollect a larger amount of virtual paint on bristles of the virtualpaintbrush; dab, swirl, or twist the paintbrush over a second area ofthe input device to virtually deposit paint of the first color from thevirtual paintbrush onto another area of the virtual palette; collectvirtual paint of a second color on the virtual paintbrush by moving thereal paintbrush over a third area of the touch sensor surface; move thereal paintbrush back to the second area to virtually mix the first paintcolor and the second paint color. In this example, the computer systemthus can generate a sequence of digital frames representing the virtualenvironment and manipulation of the virtual paintbrush and virtualpaints according to positions of the input device in real space andinputs on the touch sensor surface over time. Furthermore, once the userhas collected virtual paint on the virtual paintbrush, the user can movethe input device to a position in real space corresponding to thevirtual canvas and then contact the real paintbrush onto the inputdevice; the computer system can thus generate digitalframes—substantially in real-time—representing the virtual environmentdepicting deposition of virtual paint from the virtual paintbrush onto aregion of the virtual canvas corresponding to a position of the realpaintbrush on the touch sensor surface.

In the foregoing example application, the computer system can alsorender a virtual environment including a palette knife and canmanipulate the palette knife within the virtual environment based on theposition and orientation of the input device and a tool (e.g., a realpalette knife) on the touch sensor surface. For example, when the inputdevice is held in a position in real space corresponding to the virtualpalette, the user can manipulate the orientation of the input devicewhile drawing a real palette knife across a region of the touch sensorsurface. The computer system can then: translate changes in theorientation of the input device and the motion and force of the paletteknife across the touch sensor surface over a period of time into acorresponding sequence of three-dimensional vectors; and update thevirtual environment to depict collection of paint from the virtualpalette onto the virtual palette knife—including a twisting motion ofthe virtual palette knife as it is swept through a virtual daub of paintand then lifted away from the virtual palette knife—based on thesequence of vectors. To virtually deposit paint from the virtual paletteknife onto the virtual canvas, the user can then move the input deviceto a position in real space corresponding to the virtual canvas andplace the virtual palette knife in contact with the input devicesurface; the computer system can track these motions, as describedabove, and generate new frames representing the virtual environmentaccordingly substantially in real-time. In this example application,once the canvas is complete, the computer system can translate thecanvas into a two-dimensional print file or three-dimensionalmanufacturing file and upload the file to a printer or to an additivemanufacturing system for recreation in physical material.

The computer system can implement similar methods and techniques tocontrol the position and orientation of a scalpel within a virtualsurgical environment—such as within a virtual surgical trainingmodule—relative to a virtual human body. For example, the computersystem can modulate the virtual depth of cut of the virtual scalpel intovirtual flesh of the virtual human body with the virtual surgicalenvironment proportional to the magnitude of a force applied to thetouch sensor surface and move the virtual scalpel through virtual fleshaccording to a path traversed by the input device in real space. Thecomputer system can implement similar methods and techniques to positionand manipulate a virtual bone saw, a virtual bone drill, and/or anyother virtual surgical tool within a virtual surgical environmental.

Therefore, as in the foregoing example applications, the computer systemcan: virtually couple a virtual tool—representing the input device—to aregion of a virtual surface of a virtual object within the virtualenvironment; orient an axis of the virtual tool within the virtualenvironment to the direction of a force vector generated in Block S140according to an input on the touch sensor surface; and interface thevirtual tool with the virtual object according to the magnitude of theforce vector in Block S150. For example, the computer system can depressthe virtual tool (e.g., a virtual clay sculpting tool, as describedabove) into the virtual surface to a depth related to the magnitude ofthe force vector. The computer system can also: determine a secondposition of the input device within the real space at a second time inBlock S110; generate a second force vector including a magnitude relatedto a force magnitude of the first input moved to a second location onthe touch sensor surface and a direction related to an orientation ofthe touch sensor within the real space at the second time in Block S140;locate an origin of the second force vector within the virtualenvironment based on the second location of the first input and thesecond position of the touch sensor within the real space at the secondtime in Block S142; project the second location of the first input onthe touch sensor surface onto a second subregion of the virtual surface;orient the axis of the virtual tool to the direction of the second forcevector; draw the virtual tool across a length of the virtual surfacerelated to a distance between the first location and the secondlocation; and deform the virtual object over the length of the virtualsurface according to the magnitude of the second force vector in BlockS150. In the foregoing example, the computer system can draw the virtualtool (e.g., the virtual clay sculpting tool, as described above) throughthe object according to a trajectory of the input device and a forcemagnitude of an input on the touch sensor surface throughout thistrajectory; the computer system can then remove a corresponding virtualvolume of the virtual object or locally deform the virtual surfaceaccording to this trajectory and force magnitude(s).

However, the computer system can implement the first method S100 in anyother way for any other suitable application between any real space andvirtual environment and can interface with the input device to supportany other suitable input type or gesture, and the computer system canmanipulate the virtual surface and/or the virtual object accordingly inany other suitable way in Block S150. Furthermore, the computer systemcan execute Blocks of the first method S100 in any other suitable way tolink user inputs within physical space (e.g., on the touch sensorsurface) to the virtual environment.

1.13 Video Output

One variation of the first method S100 further includes Block S160,which recites serving a sequence of digital frames representingmanipulation of the virtual surface of the virtual object within thevirtual environment to a virtual reality headset. Generally, in BlockS160, the computer system can generate a digital frame representing astate of the virtual environment; the computer system can then servethis digital frame to a user, such as through a virtual reality headsetor through an augmented reality heads-up display. For example and asdescribed above, the computer system can generate digital framesrepresenting a user's current field of view of the virtual environmentand the current state of the virtual environment—including positions,orientations, and deformation of virtual objects and/or virtual toolsmanipulated by the user through one or more input devices—at a framerate of 6o frames per second and serve these digital frames to atwo-dimensional or three-dimensional display substantially in real-time.However, the computer system can implement any other method or techniqueto represent a state of the virtual environment in one digital image andto serve this digital image to a user substantially in real-time.

1.14 Clutching

The computer system can also implement clutching techniques to switchbetween: a first mode in which a virtual object is moved or manipulatedaccording to motion of the input device and/or forces applied to thetouch sensor surface; a second mode in which the virtual object is movedor manipulated according to a different schema when the input device ismoved and/or when a force is applied to the touch sensor surface; and athird mode in which the virtual object is not moved or manipulated whenthe input device is moved and/or when a force is applied to the touchsensor surface. For example, the computer system can switch between afirst and a second mode in which the virtual object is manipulatedaccording to two unique schema (e.g., a push schema and a pull schema, acompress schema and a bend schema) based on the total magnitude of forceapplied to the touch sensor surface. In this example, the computersystem can execute the first mode when a total force between 0.4N and0.5N is applied to the touch sensor surface, execute the second modewhen a total force greater than 0.5N is applied to the touch sensorsurface, and execute the third mode when a total force less than 0.1N isapplied to the touch sensor surface by across all inputs on the touchsensor surface.

In another example, the computing device can switch between modes basedon a number of discrete input areas represented in a touch image (e.g.,a number of finger tips in contact with the touch sensor surface). Forexample, the computing device can execute: a first mode in which a realforce on the touch sensor surface in translated into a virtual force ona virtual object when a single input area—containing the real force—isdetected on the touch sensor surface; a second mode in which thepositions of two input areas on the touch sensor surface define avirtual axis of rotation of the virtual object within the virtualenvironment and in which motion of the input device in real space istranslated into motion of the virtual object about the fixed virtualaxis of rotation of the virtual object within the virtual environmentwhen exactly two discrete input areas are detected on the touch sensorsurface; and a third mode in which the positions of three input areas onthe touch sensor surface constrain or “lock” the virtual position of thevirtual object to the real position of the input device such that thevirtual object is moved in the virtual environment according to changesto the position and orientation of the input device in real space whenexactly three discrete input areas are detected on the touch sensorsurface.

In the foregoing example, the computer system can also implement astatic or kinetic friction model (e.g., a static or kinetic frictioncoefficient for a virtual surface of a virtual object) to define astrength of a connection between an input area on the touch sensorsurface and a virtual point or virtual surface on a virtual object inthe virtual environment based on a magnitude of a force applied withinthis input area. For example, the computer system can “slip” aconnection between a virtual object and an input area on the touchsensor surface inversely with the magnitude of force applied to thetouch sensor surface within this input area according to a frictionmodel. In this example, when a force is applied to each touch sensorsurface on opposing sides of the input device to manipulate a virtualtool within the virtual environment, the computer system can slip avirtual object relative to the virtual tool according to a frictionmodel in response to a sum of a first force magnitude of a first inputon the first touch sensor surface and a second force magnitude of asecond input on the opposite touch sensor surface falling within a forcemagnitude range between an upper threshold force magnitude (in which thevirtual object is locked to the virtual tool) and a lower thresholdforce magnitude (in which the virtual object is released from thevirtual tool).

In yet another example in which the input device includes a first touchsensor surface on a first side and a second touch sensor surface on anopposing side, as described below, the computer system can execute: afirst mode in which a virtual object or virtual surface is translatedwithin the virtual environment according to a real force applied to thefirst touch sensor surface within an input area when no other input areais detected on the first or second touch sensor surface; a second modein which the virtual object is rotated within the virtual environmentabout a virtual axis defined by the positions of a first input area onthe first touch sensor surface and a second input area on the secondtouch sensor surface and by a speed or magnitude corresponding to theforce magnitudes applied to the first and second input areas when onlythe first and second input areas are detected on the first and secondtouch sensor surfaces. Alternatively, in the second mode, the computersystem can: couple a first input area on the first touch sensor surfaceto a corresponding point on one side of the virtual object based on afriction model and the force magnitude of the first input area; couple asecond input area on the second touch sensor surface to a correspondingpoint on the opposite side of the virtual object based on the frictionmodel and the force magnitude of the second input area; and apply shearforces to the virtual object as the first and second input areas moveacross the first and second touch sensor surfaces, respectively, basedon a strength of coupling between the first and second inputs areas andtheir corresponding points on the virtual object, thereby rotating thevirtual object in the virtual environment.

However, the computer system can implement any other methods andtechniques to switch between modes to control and release a virtualobject or virtual surface based on one or more inputs into the inputdevice.

2. Second Method As shown in FIG. 5, a second method includes: receivinga touch image from a handheld device, the touch image includingrepresentations of discrete inputs into a touch sensor integrated intothe handheld device in Block S230; extracting a first force magnitude ofa first input at a first location on a first side of the handheld devicefrom the touch image in Block S232; extracting a second force magnitudeof a second input at a second location on a second side of the handhelddevice from the touch image in Block S234, the second side of thehandheld device opposite the first side of the handheld device;transforming the first input and the second input into a gesture inBlock S240; assigning a magnitude to the gesture based on the firstforce magnitude in Block S242; and manipulating a virtual object withina virtual environment based on a type and the magnitude of the gesturein Block S250.

2.1 Applications

Generally, the second method can be executed by the computer systemdescribed above, such as in conjunction with the first method S100, tocharacterize multiple discrete inputs on one or more touch sensorsurfaces of an input device as a gesture and to manipulate a virtualobject within a virtual environment based on the gesture. In particular,the computer system can interface with an input device (e.g., acylindrical handheld device including a cylindrical touch sensor withcylindrical touch sensor surface, such as described below) to receiveforce magnitude and position data of multiple inputs spanning multipleplanes or distinct surfaces of the input device, to interpret theseinputs as a gesture, to define a magnitude of the gesture based on forcemagnitudes of these inputs on the touch sensor, to track the positionand orientation of the input device within the real space, and tomanipulate a virtual surface or a virtual object according to thegesture, the magnitude of the gesture, and the position of the inputdevice in real space.

For example, the computer system can interface with an input device 101including one substantially planar touch sensor on a front of the inputdevice (e.g., to face a user's thumb when held) and a secondsubstantially planar touch sensor on a back of the input device (e.g.,to face the user's fingers when held), as shown in FIG. 7.Alternatively, the computer system can interface with an input deviceincluding a single touch sensor wrapped, bent, or otherwise formed intoa three-dimensional structure, such as a cylinder or approximate sphere,spanning multiple distinct planes. The computer system can thenimplement Blocks of the second method to: identify a primary input amongmultiple discrete inputs detected on the input device at one instance intime; to detect a multi-fingered grasping gesture at the input device;to manipulate virtual fingers on a virtual hand within the virtualenvironment; to detect a bending gesture at the input device; or todetect a twisting gesture at the input device; etc.

2.2 Touch Image

Generally, in Blocks S230, S232, and S234, the computer system canimplement methods and techniques described above to: receive a touchimage from the input device (or one touch image for each discrete touchsensor integrated into the input device); to identify multiple discreteinputs on the input device from the touch image(s); and to calculate apeak force, a total force, and/or an input area for each input on theinput device at a time corresponding to the touch image(s). For example,the computer system can identify values within a touch image exceeding astatic or dynamic baseline value and label each cluster of values in thetouch image—exceeding (or exceeded by) the baseline value with aperimeter bounded by a value substantially equivalent to the baselinevalue and/or by an edge of the touch image—as a discrete input. For eachdiscrete input, the computer system can then: calculate an area of thediscrete input based on a known pitch between sense elements in theinput device or by a known area of the touch sensor and the boundary ofthe input in the touch image; identify a peak (or minimum) value withinthe boundary of the input in the touch image; and sum all values withinthe boundary of the input in the touch image to calculate a total forceof the discrete image.

Furthermore, for the input device that includes multiple touch sensorsthat each output one touch image per scan period, the computer systemcan repeat this process for each touch image corresponding to one scanperiod to identify multiple inputs across multiple touch sensor surfacesof the input device during the scan period. However, the computer systemcan implement any other methods or techniques in Blocks S230, S232, andS234 to identify discrete inputs on the touch sensor surface of theinput device at an instant in time based on a touch image received fromthe input device and corresponding to the same instant in time.

2.3 Prioritizing Inputs

In one implementation, the computer system identifies a set of distinctinputs on the input device and identifies a primary or “intended” inputin the set of inputs based on differences in force magnitudes of theseinputs. In one example, the computer system: extracts a first forcemagnitude of a first input (e.g., a thumb) at a first location on thefirst side of the input device from the touch image in Block S232;extracts a second force magnitude of a second input (e.g., an indexfinger) at a second location on the second side of the input device fromthe touch image in Block S234; and extracts a third force magnitude of athird input (e.g., a middle finger) at a third location on the secondside of the input device from the touch image in Block S234, therebyidentifying and characterizing multiple discrete inputs on the inputdevice in one instance of time. In this example, the computer systemthen: labels the first input as an intended input if the first forcemagnitude approximates the sum of the second force magnitude and thethird force magnitude and if the second force magnitude and the thirdforce magnitude are substantially similar in magnitude; label the secondinput and the third input as secondary or “holding” inputs representinginputs on the input device stemming from the user holding the inputdevice; and discard the second input and the third input accordingly.Once the first input is labeled as an intended input, the computersystem can generate a gesture corresponding to a command assigned to aregion of the handheld device including the first location, such as aselection gesture for the command, in Block S240, and write a magnitude(e.g., a degree from 1-10, a proportion from 0-100%, or a scalar value)to the gesture based on the first force magnitude in Block S242.Finally, the computer system can transform, modify, move, or otherwisemanipulate a virtual object within the virtual environment according tothe gesture according to the magnitude assigned to the gesture (e.g., byexecuting the command to a degree related to the first force magnitude).

In a similar example, the computer system can interpret a first inputapplied by an index finger to the input device as a primary inputcorresponding to pulling a virtual trigger: if the peak force magnitudeof the first input exceeds the peak force magnitude of a second inputcorresponding to the user's palm contacting an opposite side of theinput device; and if the total force magnitude of the first inputapproximates the total force magnitude of the second input. The computersystem can then move a virtual trigger on a virtual gun by a distancecorresponding to the peak force magnitude of the first input and firethe virtual gun only once the peak force magnitude of the first inputexceeds a threshold force magnitude.

However, the computer system can implement any other method, technique,or schedule to label one or multiple inputs on the input device as aprimary input, to respond to this primary input by modifying ormanipulating a virtual object within the virtual environment, and todiscard other secondary inputs on the input device.

2.4 Grasping a Virtual Object

In another implementation, the computer system implements methods andtechniques similar to those described above to interpret multiple inputson the input device as a grasping or clamping gesture. For example, thecomputer system can: transform a first input on a first side of theinput device and a second input on the opposite side of the inputdevice—represented in a touch image—into a grasping gesture based oncoaxial proximity of the first location of the first input and thesecond location of the second input through the handheld device. Inparticular, the computer system can label two (or more) inputs—such ascorresponding to a user's index finger and thumb—of similar forcemagnitudes detected on opposing sides of the input device as a graspinggesture if the first and second locations of the first and second inputsare substantially aligned in a horizontal and vertical plane passingthrough an axis of the input device, as shown in FIG. 3.

The computer system can then: write a grasping magnitude related to asum of the first force magnitude of the first input and the second forcemagnitude of the second input to the grasping gesture; couple a virtualtool in the virtual environment to the virtual object according to thegrasping magnitude of the grasping gesture, such as described above;move the virtual object—grasped by the virtual tool—throughout thevirtual environment according to motion of the input device within thereal space while the grasping magnitude exceeds a threshold magnitude;and then release the virtual object from the virtual tool once thegrasping magnitude drops below the threshold magnitude. In particular,the computer system can: track the position of the input device withinthe real space, as in Block S220 described above; move the virtualobject (and a virtual tool coupled to the virtual object according to agrasping or pinching gesture) within the virtual environment accordingto a change in the position of the input device within the real spacewhile the sum of the first force magnitude of the first input on thefirst side of the input device and the second force magnitude of thesecond input on the second side of the device exceeds a threshold forcemagnitude; and then decouple the virtual tool from the virtual objectwithin the virtual environment in response to the sum of the first forcemagnitude of the first input and the second force magnitude of thesecond input falling below the threshold force magnitude.

In one example implementation in which the input device includes a firsttouch sensor surface on one side of the input device and a second touchsensor surface on an opposite side of the input device, the computersystem can map the first touch sensor surface to a first region of avirtual surface of a virtual object represented in the virtualenvironment, and the computer system can map the second touch sensorsurface to a second region of a virtual surface of the virtual objectopposite the first region of the virtual surface. In this exampleimplementation, the input device can output a first touch image for thefirst touch sensor surface and a second touch image for the second touchsensor surface. Thus, when a pair of first and second touch images for ascan period indicate that the user is pinching the first and secondtouch sensor surfaces of the input device (e.g., between a thumb andindex finger), the computer system can compress the first and secondregions of the virtual object in the virtual environment toward oneanother based on the magnitudes of the corresponding forces on the firstand second touch sensor surfaces. In this example implementation, ifsubsequent touch image pairs indicate that the user is pinching anddragging his thumb and index finger along first and second touch sensorsurfaces of the input device, the computer system can update the firstand second regions of the virtual object in the virtual environment toreflect compression of the first and second regions toward one anotheralong a virtual arc representing the real trajectory of the user's thumband index finger across the input device. Furthermore, while the sum ofthe first and second force magnitudes exceeds a threshold forcemagnitude corresponding to a virtual weight of the virtual object, thecomputer system can update the position and orientation of the virtualobject within the virtual object environment according to the positionand orientation of the input device in real space, as described above.

2.5 Virtual Hand Control

In another implementation, the computer system implements methods andtechniques similar to those described above to interpret multiple inputson the input device as positions of virtual fingers on a virtual hand inthe virtual environment. In one example implementation in which thecomputer system interfaces with a cylindrical input device, as describedbelow, the computer system can: identify a set of discrete input areasin the touch image received from the cylindrical input device; and thenlabel each discrete input area in the set of discrete input areas as oneof a palm, a first digit, a second digit, a third digit, a fourth digit,and a fifth digit of a user's hand based on relative positions of eachdiscrete input area represented in the touch image and a digital humanhand model stored locally on the computer system in Blocks S230, S232,and S234, as shown in FIG. 5. The computer system can then generate agesture defining adduction and abduction positions of each virtualfinger of a virtual hand model based on locations and labels of eachdiscrete input area in the set of discrete input areas in Block S240;specifically, the computer system can update the lateral positions ofvirtual fingers on a virtual hand model to align to the real positionsof fingers detected on the touch sensor surface(s) of the input device.The computer system can also augment the gesture with a magnitude offlexion of each virtual finger of the virtual hand model based on aforce magnitude of each discrete input area in the set of discrete inputareas detected on the input device; specifically, the computer systemcan transform the force magnitude of each discrete input area on thetouch sensor surface labeled as representing a finger into a degree offlexion in Blocks S240 and S242. For example, the computer system candefine the position of a virtual finger in the virtual hand model: froma position of neutral extension (e.g., a straight finger) when abaseline force is represented in the touch image at an expected positionof the user's corresponding finger (or when a minimum force representingthe user's corresponding finger just grazing the touch sensor surface iscontained in the touch image); to full flexion of the virtual fingerwhen a preset maximum force is represented in the touch image at theposition of the user's finger corresponding to the virtual finger; thecomputer system can repeat this for each virtual finger of the virtualhand model. Once the computer system populates the gesture with theposition of each virtual finger for the virtual hand model based on theposition and force magnitude of each finger on the input device, thecomputer system can transform a virtual hand model within the virtualenvironment according to the adduction and abduction positions andmagnitudes of flexion defined in the gesture; specifically, the computersystem can update the adduction, abduction, flexion, and extensionpositions of each virtual finger of the virtual hand within the virtualenvironment to represent the adduction position, abduction position, andcompressive force of each of the user's fingers on the input device.

2.6 Bending the Virtual Object

In yet another implementation, the computer system implements methodsand techniques similar to those described above to interpret multipleinputs on the input device as a bending gesture to virtually bend avirtual object in the virtual environment. In one implementation inwhich the computer system interfaces with a planar input deviceincluding a first planar touch sensor on one side of the input deviceand a second planar touch sensor on an opposite side of the inputdevice, the computer system: extracts a first force magnitude of a firstinput (e.g., a thumb) at a first location on the first side of the inputdevice from the touch image in Block S232; extracts a second forcemagnitude of a second input (e.g., an index finger) at a second locationon the second side of the input device from the touch image in BlockS234; and extracts a third force magnitude of a third input (e.g., amiddle finger) at a third location on the second side of the inputdevice from the touch image in Block S234, thereby identifying andcharacterizing multiple discrete inputs on the input device in oneinstance of time, such as described above. The computer system can thentransform these three inputs into a bending gesture input according tolateral separation between the second location of the second input andthe third location of the third input on the second side of the inputdevice that spans the first location of the first input on the firstside of the input device in Block S240. The computer system can alsoassign a magnitude of a bending moment to the gesture based on the firstforce magnitude, the second force magnitude, and the third forcemagnitude, such as by defining a virtual bending moment corresponding tothe first force magnitude (which may approximate the sum of the secondforce magnitude and the third force magnitude), in Block S242. Finally,the computer system can virtually bend the virtual object in thedirection of the bending moment and to a degree corresponding to themagnitude of the bending moment as defined in the bending gesture.

In one example application in which the input device includes a firsttouch sensor surface and a second touch sensor surface, as describedabove, the computer system can detect bending and torque inputs into theinput device based on touch image pairs for the first and second touchsensor surfaces and modify the virtual object accordingly. For example,the computer system can: correlate a first force applied near the centerof the first touch sensor surface, a second force applied near a rightedge of the second touch sensor surface, and a third force applied neara left edge of the second touch sensor surface with a concave bendinginput; map a plane in the real space parallel to the first and secondtouch sensor surfaces onto a virtual reference plane intersecting thevirtual object within the virtual environment based on the position andorientation of the input device; and deform the virtual object out ofthe virtual reference plane according to the magnitude of the concavebending input. In another example, the computer system can: correlate afirst force applied near a top-right corner of the input device at thefirst touch sensor surface, a second force applied near the bottom-rightcorner of the input device at the second touch sensor surface, a thirdforce applied near a bottom-left corner of the input device at the firsttouch sensor surface, a second force applied near the top-left corner ofthe input device at the second touch sensor surface with a torque input(or a “twisting gesture”) about a lateral axis of the input device; mapthe lateral axis onto a virtual axis intersecting the virtual objectwithin the virtual environment based on the position and orientation ofthe input device; and twist the virtual object (e.g., a virtual wet rag)about the virtual axis according to the magnitude of the torque input,as shown in FIG. 4.

In a similar example application, the computer system interfaces withthe input device to control a virtual Rubik's cube. In particular, inthis example application, the computer system generates digital framesof a virtual brick model and virtual bricks in a virtual brick pile. Forexample, the user can rotate the input device about three axes, and thecomputer system can correlate the orientation of the input device withone of three dimensions of the virtual Rubik's cube for subsequentmanipulation. The user can then translate the input device between threediscrete position ranges along the axis with which the input device iscurrently oriented, and the computer system can correlate the positionof the input device along the selected axis with one of three virtualrows of the virtual Rubik's cube for subsequent manipulation. The usercan then place two (or more) fingers on the touch sensor surface andthen rotate these fingers over the touch sensor surface; the computersystem can map this input motion into rotation of the selected virtualrow of the virtual Rubik's cube while holding the remaining two rows ofthe virtual Rubik's cube static within the virtual environment. Thecomputer system can update the virtual environment accordinglysubstantially in real-time.

2.7 Detecting Inputs on Unsensed Areas of the Input Device

In another implementation, the computer system implements methods andtechniques similar to those described above to transform inputs on atouch sensor surface represented in a touch image into determination ofan input (e.g., a finger depressing) contacting an unsensed surface(i.e., a surface not containing a touch sensor, moment switch, or othercontact or proximity sensor) of the input device. For example, when auser holding a cylindrical input device between his palm and fourfingers depresses the top of the cylindrical input device—excluding atouch sensor or other sensor—with a thumb, the user may also grasp or“squeeze” the cylindrical input device between his thumb and fingers toa greater degree to prevent the input device from slipping from hisgrasp. However, depression of the top surface of the input device maystill cause the input device to shift slightly downward in the user'shand, which may cause contact areas of the user's fingers on the touchsensor surface to move (or “roll”) upward relative to the touch sensorsurface. The grasping force applied by the user may then decrease andthe position of the user's fingers on the touch sensor surface mayreturn to previous lower relative positions once the user's releases histhumb from the top of the input device. The computer system can thusinterpret this sequence—including increases in peak or total forcemagnitude applied by each finger to the touch sensor surface and upwardshifts in input area positions followed by decreases in peak or totalforce magnitude applied by the user's fingers and downward shifts ininput area positions on the touch sensor surface—as an input on the topof the input device.

Therefore, as in the foregoing example implementation in which the inputdevice defines a cylindrical (or otherwise three-dimensional) touchsensor surface, the computer system can: track a first input on a firstside of the input device and a second input on a second side of theinput device across a sequence of touch images in Blocks S230 and S232;and then interpret increases in force magnitudes of the first input andthe second input, an upward shift in the first location of the firstinput, and an upward shift in the second location of the second inputacross the sequence of touch images as a third input on a unsensedsurface of the input device above and substantially perpendicular to thefirst side and to the second side of the handheld device. For example,the computer system can implement blob detection techniques to identifyone discrete input area in one touch image and to match this discreteinput area in a subsequent touch image, calculate the centroid of theinput area in each touch image, calculate a distance between centroidsof the discrete input area across the two touch images, and repeat thisprocess over subsequent touch images to track movement of thecorresponding input on the input device over time. The computer systemcan also implement methods and techniques described above to calculate apeak or total force magnitude on the discrete input area over thesequence of touch images and then implement an unsensed area input modelor template to characterize this sequence of force magnitude change andinput area location change events as a single input on the unsensed topof the input device.

In the foregoing implementation, the computer system can also: label thethird input on the unsensed top of the input device as an intended inputon the input device, discard the first input and the second inputrepresented in the sequence of touch images, and generate a gesturecorresponding to a command assigned to the top unsensed area of theinput device in Block S240; and assign a third force magnitude to thethird input on the unsensed top of the input device based on the forcemagnitudes of the first input and the second input, a magnitude of theupward shift in the first location of the first input, and/or amagnitude of the upward shift in the second location of the second inputin Block S242. For example, the computer system can relate the totalforce magnitudes of the first and second inputs on the touch sensorsurface and the maximum displacement distance of the first or secondinput during application of the user's thumb on the top of the inputdevice to a total force magnitude of the user's thumb on the top of theinput device. Finally, the computer system can manipulate the virtualobject within the virtual environment according to the location andforce magnitude of this input on the input device. For example, thecomputer system can select the virtual object within the virtualenvironment according to the command assigned to the top of the inputdevice if the estimated force magnitude of this input exceeds athreshold force magnitude.

3. Input Device

As shown in FIGS. 7, 8, and 9, a system 100 for manipulating virtualobjects within a virtual environment includes: a housing 102; a firsttouch sensor 110; a second touch sensor 120; and a controller 130. Thefirst touch sensor includes: a first touch sensor surface 118 extendingacross a first region of the exterior of the housing 102; a first arrayof sense electrodes 114 patterned across a substrate 112; and a firstresistive layer 116 interposed between the first touch sensor surface118 and the first array of sense electrodes 114, in contact with thefirst array of sense electrodes 114, and including a material exhibitingvariations in local electrical resistance responsive to variations inforce applied to the first touch sensor surface 118. The second touchsensor 120 includes: a second touch sensor surface 128 extending acrossa second region of the exterior of the housing 102 opposite the firstregion; a second array of sense electrodes 122; and a second resistivelayer 126 interposed between the second touch sensor surface 128 and thesecond array of sense electrodes 122, in contact with the second arrayof sense electrodes 122, and including the material. The controller 130is coupled to the housing and is configured to: scan the first array ofsense electrodes 114 and the second array of sense electrodes 122 duringa scan period; detect a first input of a first force magnitude at afirst location on the first touch sensor surface 118 based on a changein resistance values of a subset of sense electrodes in the first arrayof sense electrodes 114; detect a second input of a second forcemagnitude at a second location on the second touch sensor surface 128based on a change in resistance values of a subset of sense electrodesin the second array of sense electrodes 122; and generate a touch imagedefining the first location and the first magnitude of the first inputand the second location and the second magnitude of the second input forthe scan period.

3.1 Applications

Generally, the system 100 includes an input device, as described above,that includes one or more touch sensors that detect positions and forcemagnitudes of inputs on one or more touch sensor surfaces, that packagesthe positions and force magnitudes of inputs on the touch sensorsurface(s) into one touch image per scan period; and that uploads thesetouch images to a computer system, such as described above, forprocessing to manipulate virtual objects within a virtual environment.For example, the system 100 (hereinafter “input device”) can define ahandheld device that a user may depress, squeeze, twist, bend, shake,move, or otherwise manipulate in real space with one or two hands, andthe input device 101 can package inputs on the touch sensor surface(s)into touch images and upload these touch images to a local virtualreality system—including the computer system—substantially in real-time.In this example, the computer system can then implement the first methodS100 and/or the second method S200 described above to track the inputdevice 101 in real space, to define gestures based on the position ofthe input device 101 in real space and inputs represented in one or asequence of touch images, and to manipulate virtual objects within avirtual environment according to such gestures.

3.2 Touch Sensor

As shown in FIGS. 7, 8, and 9, the input device 101 includes one or moretouch sensors, wherein each touch sensor includes: an array of senseelectrode and drive electrode pairs (i.e., sense electrodes, or “senseelements”) patterned across a substrate (e.g., a fiberglass PCB, aflexible PCB); and a resistive layer arranged over the substrate incontact with the sense electrode and drive electrode pairs, defining amaterial exhibiting variations in local bulk resistance and/or localcontact resistance responsive to variations in applied force, anddefining a touch sensor surface opposite the substrate. As described inU.S. patent application Ser. No. 14/499,001, the resistive touch sensorcan include a grid of drive electrodes and sense electrodes patternedacross the substrate. The resistive layer can span gaps between eachdrive and sense electrode pair across the substrate such that, when alocalized force is applied to the touch sensor surface, the resistanceacross an adjacent drive and sense electrode pair varies proportionally(e.g., linearly, inversely, quadratically, or otherwise) with themagnitude of the applied force. As described below, the controller canread resistance values across each drive and sense electrode pair withinthe touch sensor and can transform these resistance values into aposition and magnitude of one or more discrete force inputs applied tothe touch sensor surface.

3.3 Controller

The input device 101 also includes a controller 130 that detects thefirst input on the first touch sensor surface based on a change inmeasured resistance across sense electrodes and drive electrodes in thesubset of sense electrode and drive electrode pairs and determines aforce magnitude of the first input based on a magnitude of the change inmeasured resistance across sense electrode and drive electrodes in thesubset of sense electrode and drive electrode pairs. Generally, thecontroller functions to drive the touch sensor, to read resistancevalues between drive and sense electrodes during a scan cycle, totransform resistance data from the touch sensor into locations andmagnitudes of force inputs over the touch sensor surface, and to packagethese data in one touch image per scan cycle, as shown in FIGS. 1 and3-7. The controller can also execute Blocks of the first method S100 andthe second method S200 described above locally to transform locationsand/or force magnitudes of inputs on the touch sensor(s) detected overone or more scan cycles into a gesture or other command and to outputvalues of this gesture or command to a local computer system; thecomputer system can then execute other Blocks of the first method S100and the second method S200 described above to manipulate a virtualobject within a virtual environment according to this gesture orcommand.

In one implementation, the controller includes: an array column driver(ACD); a column switching register (CSR); a column driving source (CDS);an array row sensor (ARS); a row switching register (RSR); and an analogto digital converter (ADC); as described in U.S. patent application Ser.No. 14/499,001. In this implementation, the touch sensor can include avariable impedance array (VIA) that defines: interlinked impedancecolumns (IIC) coupled to the ACD; and interlinked impedance rows (IIR)coupled to the ARS. During a resistance scan period: the ACD can selectthe IIC through the CSR and electrically drive the IIC with the CDS; theVIA can convey current from the driven IIC to the IIC sensed by the ARS;the ARS can select the IIR within the touch sensor and electricallysense the IIR state through the RSR; and the controller can interpolatesensed current/voltage signals from the ARS to achieve substantiallyaccurate detection of proximity, contact, pressure, and/or spatiallocation of a discrete force input over the touch sensor for theresistance scan period within a single sampling period.

3.4 Optical Fiducials and Wireless Transmitter

In one variation shown in FIG. 8, the input device 101 also includes anoptical emitter 140 coupled to the housing and configured to broadcastan optical signal detectable by an external tracking system to determinethe position and orientation of the housing within a real space. Inparticular, the input device 101 can include a passive or active opticalfiducial 140, such as an infrared reflector or optical emitter 140, thatinterfaces with an external tracking system to enable the computersystem to track the position of the input device 101 in real space, asdescribed above. (Alternatively, this input device 101 can include anoptical detector and can track its position and orientation in realspace based on infrared or other light signals received from staticoptical emitters placed in the real space and then upload its positionand orientation to the computer system, such as via a wirelesstransmitter 150 described below.)

In an implementation in which the input device 101 is a wireless,self-contained handheld device that interfaces with a remote computersystem to control a virtual environment, the input device 101 can alsoinclude a wireless transmitter 150 (or wireless transceiver) arrangedwithin the housing and configured to transmit touch images to thecomputer system, such as over short-range wireless communicationprotocol, as shown in FIG. 8. Alternatively, the input device 101 can bewired to the computer system and can upload touch images to the computersystem substantially in real-time over wired communication protocol. Asdescribed above, the input device 101 can further include one or moremotion sensors, such as an accelerometer, gyroscope, and/ormagnetometer, and can upload motion data to the computer system witheach touch image; the computer system can then manipulate these sensordata to track the input device 101 in real space over time.

3.5 Single-Faceted Input Device In one variation, the housing defines arectangular body, and the input device 101 includes a single touchsensor no defining a planar touch sensor surface 118 on one side of thehousing 102. For example, the housing 102 can include a rectilinear,plastic injection-molded body or a machined aluminum housing. Similarly,the input device 101 can include a single touch sensor 110 integratedinto an ergonomic housing 102, as shown in FIG. 6.

3.6 Multi-Faceted Input Device

In another variation shown in FIG. 7, the housing 102 defines arectilinear body and the input device 101 includes two touch sensors,including a first touch sensor defining a first planar touch sensorsurface across the front (or top) of the housing 102 and a second touchsensor defining a second planar touch sensor surface across the bottomfront (or back) of the housing 102 opposite the first touch sensorsurface. In this variation, the first and second touch sensors can bediscrete, wherein each touch sensor includes its own discrete substrateand is paired with its own dedicated controller.

In a similar variation, the input device 101 includes one touch sensorthat defines two touch sensor surfaces each spanning a unique plane ofthe housing 102. For example, the input device can include a touchsensor fabricated on a flexible PCB, and the touch sensor can be bent,folded, or otherwise formed into a three-dimensional structure that,when installed in the housing 102, spans two opposing sides (and one ormore intermediate sides) of the housing 102, such as the front and back(and one or both sides) of the housing 102. For example, substrate ofthe touch sensor can include a cross-shaped flexible PCB that can befolded into a three-dimensional rectilinear structure and installed overfive of six planar sides of the housing 102 and fastened to these sidesof the housing 102, such as with an adhesive or mechanical fastener; aresistive layer, such as in the form of a sleeve that can then beinstalled over the flexible PCB and the housing 102 to complete thetouch sensor. A cover layer can be integrated into the sleeve over theresistive layer to define a continuous touch sensor surface across thefive sensible sides of the housing 102.

The controller 130, the transmitter 150, and a battery 160 can then beinstalled in the housing 102 via the sixth side (e.g., the bottom) ofthe housing 102 to complete the input device 101. The input device canthus include one controller that outputs one touch image representinglocations and force magnitudes of inputs on the five sensible sides ofthe housing 102 per scan cycle. The computer system can thus implement amodel of sense element locations across the sides of the housing 102 topartition a two-dimensional touch image (e.g., a two-dimensional matrix)into one discrete touch image per side of the three-dimensional housing.Furthermore, upon receipt of a sequence of touch images from the inputdevice 101, the computer system can execute Blocks of the second methodS200 described above to identify and characterize inputs on the sixthside of the input device 101 based on locations and force magnitudes ofinputs on the five sensible sides of the housing 102 represented in thesequence of touch images. However, the touch sensor can be fabricatedand formed into a three-dimensional structure that spans any other twoor more sides of the housing 102 to define multiple touch sensorsurfaces spanning multiple unique planes of the housing 102.

3.6 Cylindrical Input Device

In another variation shown in FIG. 8, the housing 102 defines acylindrical section, and the first touch sensor surface and the secondtouch sensor surface define a contiguous touch sensor surface spanning acylindrical surface of the housing 102. Generally, in this variation,the housing 102 defines a cylindrical section, and the touch sensordefines a cylindrical touch sensor surface extending over all or aportion of the cylindrical section of the housing 102. For example, thehousing 102 can include a rigid cylindrical section—such as a plasticinjection-molded, aluminum die cast, or extruded aluminum tube—open onone or both ends. As in the foregoing variation, the substrate of thetouch sensor can include a flexible PCB; once an array of drive andsense electrodes are patterned across the flexible PCB, the resistivelayer can be applied over the array of drive and sense electrodes, and acover layer can be applied over the resistive layer to form a closed,self-contained touch sensor. The touch sensor can then be wrapped aroundand fixed to the housing 102, such as with an adhesive or mechanicalfastener. Alternatively, the substrate, array of drive and senseelectrodes, and resistive layer can be installed over the housing 102,and the housing 102 and touch sensor assembly can be stuffed into acover sleeve, such as a flexible silicone sleeve or a heat-shrink tube,to close the touch sensor and to fix the touch sensor to the housing102. Similarly, the resistive layer can be integrated into the cloversleeve, as described above. The controller 130, the transmitter 150, anda battery 160 can then be installed in one open end of the housing 102and closed with a cover to complete the input device 101.

Therefore, the input device 101 can include multiple touch sensorsections (e.g., columns of sense elements that span the height of thehousing 102) fabricated on one common substrate that can be formedaround multiple sides of a housing or across a non-planar (e.g.,cylindrical) surface of a housing, and these multiple touch sensorsections can share one common controller.

3.7 Spherical Input Device

In yet another variation shown in FIG. 9, the input device 101 defines aspherical form. For example, the substrate of the touch sensor caninclude a flexible PCB defining a perimeter approximating atwo-dimensional projection of the surface of a three-dimensional sphere;and an array of sense elements can be patterned across the flexiblesubstrate within the bounds of this two-dimensional projection of thesurface of the three-dimensional sphere. In this example, the housing102 can define an approximately spherical, rigid structure, and thesubstrate can be folded around and fixed to the housing 102. In thisexample, gaps between the housing 102 and the back side of the substratecan be potted, such as with an epoxy or urethane, to rigidly mount thesubstrate to the housing 102. A flexible resistive layer and cover layercan then be wrapped over the substrate to complete the touch sensor.Similarly, the flexible resistive layer can be wrapped over and fixed tothe substrate, and the cover layer can be overmolded (e.g., in silicone)or adhered over the flexible resistive layer and substrate to form asubstantially continuous spherical surface. Alternatively, the resistivelayer and the substrate can be assembled prior to installation of thetouch sensor over the housing 102, as described above.

In this implementation, the housing 102 can also include two hollow,hemispherical sections, and the input device 101 can include twodiscrete touch sensors, such as: one touch sensor installed over one ofthe hemispherical sections and defining a first touch sensor surface;and a second touch sensor installed over the other hemispherical sectionand defining a second touch sensor surface. The controller 130, thetransmitter 150, a battery 160, and/or other communication, control,processing, or power elements can then be installed in one or both ofthe hemispherical sections, and the hemispherical sections can beassembled to form one spherical input device.

However, the input device 101 can define any other form and can includeany other number of touch sensors defining one or more discrete touchsensor surfaces across the input device 101.

3.8 Overlays

One variation of the system 100 further includes a sleeve 170 or“overlay” defining a receptacle configured to transiently receive theinput device 101, defining an external surface beyond the receptacle,and configured to communicate a force on its external surface into thetouch sensor surface(s) of the input when the input device 101 isinstalled in the receptacle, as shown in FIGS. 8 and 10. Generally, inthis variation, the system 100 can define a kit, including the inputdevice 101 and one or more overlays that can be installed over the inputdevice 101 to modify an effective form factor of the input device 101.However, each overlay can be of or include a flexible material thatcommunicates forces applied to the exterior surface(s) of the overlayinto the touch sensor surface(s) of the input device 101, therebyenabling the touch sensor(s) in the input device 101 to detect inputsapplied to the exterior surface(s) of the overlay and to represent theseinputs in touch images despite a physical offset between the touchsensor surface(s) of the input device 101 and the exterior surface(s) ofthe overlay. For example, an overlay can include a flexible silicone,urethane, foam (e.g., silicone foam, latex foam), or textile (e.g.,leather) body defining a receptacle into which a rectilinear,cylindrical, or spherical, input device can be inserted. The system 100can thus define a kit including an input device and a set of sleeves oroverlays, wherein the input device 101 can be operated by a userindependently of a sleeve or can be inserted into any one of varioussleeves in the kit to alter the form of the input device 101, such as tomatch the form of the input device 101 to a virtual tool shown within avirtual environment as the user plays a virtual reality game.

In one example, the system 100 includes a sleeve defining a handedergonomic overlay including a molded foam body forming an internalreceptacle for the input device 101 and a curvilinear exterior gripsurface configured to mate with a human's right (or left) hand, as shownin FIG. 8. In another example, the system 100 includes a sleeve defininga handgun overlay including a body molded in a low-durometer siliconeinto the form of a handgun with grip, receiver, hammer, and trigger,trigger guard, and barrel and forming an internal receptacle for theinput device 101 extending from proximal the barrel, through thereceiver, near the trigger, and to the grip, as shown in FIG. 10. Inthis example, the handgun overlay can communicate a force applied to thesurface near the hammer into one side of the input device 101; the inputdevice 101 can represent this force in a touch image and transmit thistouch image to the computer system; and the computer system can map thelocation of this input represented in the touch image to a virtualhammer on a virtual handgun and cock the virtual hammer (or release thevirtual hammer depending on a direction of the input over a sequence oftouch images) if the force magnitude of this input exceeds a thresholdhammer cocking force magnitude. In this example, the handgun overlay canalso communicate a force applied to the surface near the trigger intothe opposite side of the input device 101; the input device 101 canrepresent this force in a second touch image and transmit this secondtouch image to the computer system; and the computer system can map thelocation of this input represented in the second touch image to avirtual trigger on the virtual handgun and pull the virtual trigger ofthe virtual gun in the virtual environment if the force magnitude ofthis input exceeds a threshold trigger pull force magnitude. In thisexample, a sleeve can therefore define a form of a physical toolrepresenting a virtual tool shown within a virtual environment.

In yet another example, the system 100 includes a sleeve defining ahandheld gaming controller overlay including a flexible foam body withrigid (e.g., urethane) inserts 172, as described below, defining falsebutton areas (described below), a D-pad, and a joystick over top andfront surfaces of the handheld gaming controller overlay. When each ofthe false button areas is depressed, the D-pad pivoted, or the joystickmanipulated by a user when the input device 101 is installed in thehandheld gaming controller overlay, the corresponding rigid inserts 172can move within the flexible foam body and transfer forces appliedthereto onto corresponding regions of the touch sensor surface(s) of theinput device 101. The input device 101 can represent such forces appliedto the rigid elements in touch images; and the computer system can maplocations of these inputs represented in these touch images to commandsassociated with each of these false button areas, D-pad direction, andjoystick positions and execute these commands accordingly within thevirtual environment substantially in real-time.

However, the system 100 can include one or more sleeves defining anyother form and of any other material.

As described above, the exterior surface of the sleeve 170 can alsodefine a false button area 190, such as cylindrical emboss or debosstactilely representative of a physical button, as shown in FIG. 8. Thuswhen the input device 101 is installed in the sleeve 170 and a falsebutton area on the sleeve 170 is depressed by a user's finger, the bodyof the sleeve 170 can transfer the force of this input into the touchsensor of the input device 101 in a direction substantially normal tothe touch sensor surface, and the input device 101 can generate a touchimage representing the force magnitude and location of this input. Uponreceipt of this touch image from the input device 101, the computersystem can map the location of the input represented in the touch imageto a command associated with the false button area and execute thiscommand if the force magnitude of the input represented in the touchimage exceeds a threshold force magnitude characteristic of a physicalbutton (e.g., a physical snap button). For example, the input device 101can include an alignment feature (e.g., a tab or a magnet), and thesleeve 170 can define an alignment receiver (e.g., a groove or a ferrouselement) configured to mate with the alignment feature of the inputdevice 101, thereby constraining the input device 101 to a particularorientation within the sleeve 170 when installed in the receptacle ofthe sleeve 170. In this example, the computer system can implement avirtual map of false button areas on the sleeve 170 to match an inputrepresented in a touch image to a command associated with such falsebutton areas.

In another example, the sleeve 170 can include one or more ridges,knobs, or other substantially rigid features that extend into thereceptacle to contact and depress the touch sensor surface of the inputdevice 101 when the input device 101 is installed in the receptacle. Thecontroller can represent forces applied to the touch sensor surface bythese features in a touch image; and the computer system can identifyvalues in the touch image representing these features; determine anorientation of the input device 101 within the sleeve 170, and locate avirtual map of false button areas on the sleeve 170 to the touch image(and subsequent touch images) received from the input device 101accordingly. In this example, the sleeve 170 can include a substantiallyunique array of such features, and the computer system can also identifythe type of sleeve containing the input device 101 based on the array offorces applied by these features represented in a touch image receivedfrom the input device 101. Similarly, as in the foregoing example, thesleeve 170 can include one or more magnets that function as alignmentfeatures; the input device 101 can include ferrous element and magneticfield (e.g., Hall-effect) sensor pairs that magnetically couple to andthat detect proximity to these magnetic elements, respectively; and theinput device 101 or the computer system can identify the sleeve 170 inwhich the input device 101 is installed based on a presence of magneticfields detected by each magnetic field sensor in the input device 101.Alternatively, the sleeve 170 can include an RFID tag that wirelesslybroadcasts an identifier (e.g., a UUID), an ID chip that outputs anidentifier over wired communication protocol, or any other wired orwireless ID mechanism; the input device 101 can collect an identifierfrom the sleeve 170 and pass this to the computer system; and thecomputer system can identify the sleeve 170 and then implement acorresponding virtual map of false button areas on the sleeve 170 andcorresponding commands and threshold force magnitudes based on anidentifier received from the sleeve 170. Yet alternatively, the computersystem can prompt the user to manually identify whether the input device101 has been installed in the sleeve 170 and/or which of variousavailable sleeves is currently in use.

As described above, a sleeve can also include: a rigid insert 172 of afirst durometer; and a flexible substrate of a second durometer lessthan the first durometer, interposed between the receptacle and therigid insert 172, configured to deform in response to application of aforce on the rigid insert 172 toward the receptacle, and configured tocommunicate a force applied to the rigid insert 172 into the first touchsensor surface, as shown in FIG. 10. Generally, the sleeve 170 caninclude elements of different rigidity (e.g., durometer) that cooperateto deform and to communicate forces on the surface of the sleeve 170into a touch sensor surface of the input device 101. For example, asleeve can include only a moderately flexible material (e.g., amedium-durometer silicone) in volumes of the sleeve 170—between theexterior surface of the sleeve 170 and a touch sensor surface on theinput device 101—less than five millimeters in thickness. In thisexample, the sleeve 170 can also include a moderately flexible material(e.g., a medium-durometer solid silicone) over a more flexible material(e.g., a low-durometer silicone foam) in volumes of the sleeve170—between the exterior surface of the sleeve 170 and a touch sensorsurface on the input device 101—between five millimeters and tenmillimeters in thickness. However, for volumes of the sleeve 170—betweenareas of interest on the surface of the sleeve 170 (e.g., a false buttonarea, a false trigger for a handgun overlay, a false D-pad, a falsejoystick, etc.) and a nearest area on the touch sensor surface of theinput device 101—exceeding ten millimeters, the sleeve 170 can include arigid insert 172 (e.g., a high-durometer silicone beam or pillar)extending from the area of interest on the surface of the sleeve 170 tothe interior surface of the receptacle for the input device 101.Alternatively, in this implementation, the sleeve 170 can include arigid insert 172 extending from the area of interest on the surface ofthe sleeve 170 to a depth offset from the receptacle (e.g., by adistance of two millimeters) and a flexible volume (e.g., an expandedsilicone foam insert) between the end of the rigid insert 172 and thereceptacle for the input device 101.

For example, a handheld gun overlay can include a rigid triggersuspended from a flexible member or living hinge, as shown in FIG. 10.In another example, a handheld gaming controller overlay can include arigid false button, a rigid D-pad insert, and a rigid joystick insertsuspended from the body of the overlay with a soft silicone member. Whendepressed or manipulated by a user holding the sleeve 170 and inputdevice assembly, the rigid insert 172 can communicate an applied forceinto the adjacent touch sensor of the input device 101, and the inputdevice 101 can represent this applied force in a touch image. However, asleeve can include any other rigid or flexible sections, volumes, orelements that cooperate to communicate forces on the surface of thesleeve 170 into one or more touch sensors of the input device 101.

A sleeve can further include passive optical fiducials or windows thatalign with active optical fiducials arranged on the input device 101 inorder to enable an external system to track the position of the inputdevice 101 and sleeve assembly in real space when in use.

3.9 Strap

One variation of the input device 101 further includes a strap 180extending from the housing 102 and configured to couple the housing 102to a user's hand. For example, the input device 101 can include a softstrap configured to wrap around a user's wrist, as shown in FIG. 8. Inanother example, the strap 180 includes a rigid member that engages auser's forearm and/or palm and functions to constrain the input device101 (and installed sleeve) proximal the user's palm, as shown in FIG. 9.When worn, the strap 180 can thus enable a user to release the inputdevice 101 with his hand without dropping the input device 101; thecomputer system thus interprets withdrawal of all inputs on touchsensor(s) as release of the input device 101 and updates a virtualobject or virtual tool within the virtual environment accordingly. Withthe strap 180 holding the input device 101 at the user's fingertips, theuser can then later close his hand around the input device 101, such asto grasp a virtual object or virtual tool within the virtualenvironment.

4. Integrated Device

In one variation, the first method S100 and/or the second S200 areexecuted by a singular (e.g., unitary) device including a computersystem, a digital display, a transparent (or translucent) touch sensorsurface arranged over the digital display. For example, the device caninclude a pressure-sensitive touch sensor-enabled tablet. In thisvariation, the device can execute Blocks of the method: to track its ownposition and orientation (and therefore the position and orientation ofthe touch sensor surface); to record magnitudes and locations of forceson the touch sensor surface; to generate digital frames of a virtualenvironment including a virtual surface of a virtual object manipulatedaccording to the position and orientation of the computing device andthe location and magnitude of a force on the touch sensor surface; andto render these digital frames on the integrated digital displaysubstantially in real-time. In a similar variation, the method isexecuted by a singular device including a computer system, a touchsensor surface, and a digital display arranged over the touch sensorsurface. However, Blocks of the method can be executed by any otherdevice or combination of devices.

The systems and methods of the embodiments can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A system for manipulating virtual objects within a virtualenvironment comprising: a housing; a first touch sensor area comprising:a first touch sensor surface extending across a first region of theexterior of the housing; a first array of sense electrodes patternedacross a substrate; and a first resistive layer interposed between thefirst touch sensor surface and the first array of sense electrodes, incontact with the first array of sense electrodes, and comprising amaterial exhibiting variations in local electrical resistance responsiveto variations in force applied to the first touch sensor surface; asecond touch sensor area comprising: a second touch sensor surfaceextending across a second region of the exterior of the housing oppositethe first region; a second array of sense electrodes; and a secondresistive layer interposed between the second touch sensor surface andthe second array of sense electrodes, in contact with the second arrayof sense electrodes, and comprising the material; a controller coupledto the housing and configured: to scan the first array of senseelectrodes and the second array of sense electrodes during a scanperiod; to detect a first input of a first force magnitude at a firstlocation on the first touch sensor surface based on a change inresistance values of a subset of sense electrodes in the first array ofsense electrodes; to detect a second input of a second force magnitudeat a second location on the second touch sensor surface based on achange in resistance values of a subset of sense electrodes in thesecond array of sense electrodes; and to generate a touch image definingthe first location and the first magnitude of the first input and thesecond location and the second magnitude of the second input for thescan period.
 2. The system of claim 1: wherein the first array of senseelectrodes comprises a set of sense electrode and drive electrode pairs;wherein the first resistive layer exhibits variations in electricalcontact resistance over a subset of sense electrode and drive electrodepairs responsive to application of a force to the first touch sensorsurface over the subset of sense electrode and drive electrode pairs;and wherein the controller detects the first input on the first touchsensor surface based on a change in measured resistance across senseelectrode and drive electrodes in the subset of sense electrode anddrive electrode pairs and determines a force magnitude of the firstinput based on a magnitude of the change in measured resistance acrosssense electrode and drive electrodes in the subset of sense electrodeand drive electrode pairs.
 3. The system of claim 1, wherein the housingdefines a cylindrical section; and wherein the first touch sensorsurface and the second touch sensor surface comprise a contiguous touchsensor surface spanning a cylindrical surface of the housing.
 4. Thesystem of claim 1, wherein the first touch sensor surface defines afirst planar area on a first side of the housing, and wherein the secondtouch sensor surface defines a second planar area on a second side ofthe housing opposite the first side of the housing.
 5. The system ofclaim 1, further comprising a computer system configured: to receive asequence of touch images corresponding to a sequence of sampling periodsfrom the controller; to track the first input and the second inputacross the sequence of touch images; and to interpret increases in forcemagnitudes of the first input and the second input, an upward shift inthe first location of the first input, and an upward shift in the secondlocation of the second input across the sequence of touch images as athird input on a unsensed surface of the housing above and substantiallyperpendicular to the first touch sensor surface and to the second touchsensor surface.
 6. The system of claim 5, wherein the computer system isremote from the housing; and further comprising a transmitter arrangedwithin the housing and configured to transmit the sequence of touchimages to the computer system.
 7. The system of claim 5, wherein thecomputer system estimates a third force magnitude of the third input onthe unsensed surface of the housing based on the force magnitudes of thefirst input and the second input, a magnitude of the upward shift in thefirst location of the first input, and a magnitude of the upward shiftin the second location of the second input.
 8. The system of claim 1,further comprising a sleeve defining a receptacle configured totransiently receive the housing, defining an external surface beyond thereceptacle, and configured to communicate a force on the externalsurface into the first touch sensor surface when the housing isinstalled in the receptacle.
 9. The system of claim 8, wherein thesleeve defines a handed ergonomic overlay; and further comprising asecond sleeve defining a second receptacle configured to transientlyreceive the housing, defining a second external surface beyond thesecond receptacle, configured to communicate a force on the secondexternal surface into the first touch sensor surface when the housing isinstalled in the second receptacle, and defining a form of a toolcorresponding to a virtual tool within a virtual environment.
 10. Thesystem of claim 8, wherein the sleeve comprises: a rigid insert of afirst durometer; and a flexible substrate of a second durometer lessthan the first durometer, interposed between the receptacle and therigid insert, configured to deform in response to application of a forceon the rigid insert toward the receptacle, and configured to communicatea force applied to the rigid insert into the first touch sensor surface.11. system of claim 1, wherein the first touch sensor area comprises thefirst array of sense electrodes and the second touch sensor areacomprises the second array of sense electrodes patterned across thesubstrate; and wherein the substrate is assembled into athree-dimensional structure within the housing to face the first touchsensor surface and the second touch sensor surface.
 12. The system ofclaim 1, further comprising a strap extending from the housing andconfigured to couple the housing to a hand of a user.
 13. The system ofclaim 1, further comprising an optical emitter coupled to the housingand configured to broadcast an optical signal detectable by an externaltracking system to determine the position and orientation of the housingwithin a real space.
 14. A method for manipulating virtual objectswithin a virtual environment comprising: receiving a touch image from ahandheld device, the touch image comprising representations of discreteinputs into a touch sensor integrated into the handheld device;extracting a first force magnitude of a first input at a first locationon a first side of the handheld device from the touch image; extractinga second force magnitude of a second input at a second location on asecond side of the handheld device from the touch image, the second sideof the handheld device opposite the first side of the handheld device;transforming the first input and the second input into a gesture;assigning a magnitude to the gesture based on the first force magnitude;and manipulating a virtual object within a virtual environment based ona type and the magnitude of the gesture.
 15. The system of claim 14:further comprising extracting a third force magnitude of a third inputat a third location on the second side of the handheld device from thetouch image; wherein transforming the first input and the second inputinto the gesture comprises: labeling the first input as an intendedinput based on the first force magnitude approximates a sum of thesecond force magnitude and the third force magnitude; discarding thesecond input and the third input; and generating the gesturecorresponding to a command assigned to a region of the handheld devicecomprising the first location; and wherein manipulating the virtualobject within the virtual environment comprises transforming the virtualobject according to the command to a degree related to the first forcemagnitude.
 16. The system of claim 14, wherein extracting the firstforce magnitude and the first location of the first input and the secondforce magnitude and the second location of the second input comprisetracking the first input and the second input across a sequence of touchimages; wherein transforming the first input and the second input into agesture comprises interpreting increases in force magnitudes of thefirst input and the second input, an upward shift in the first locationof the first input, and an upward shift in the second location of thesecond input as a third input on a unsensed surface of the handhelddevice above and substantially perpendicular to the first side and tothe second side of the handheld device; labeling the third input as anintended input on the handheld device; discarding the first input andthe second input; and generating the gesture corresponding to a commandassigned to a region of the handheld device comprising the unsensedsurface; wherein assigning the magnitude to the gesture comprisesassigning a third force magnitude to the third input on the unsensedsurface of the housing based on the force magnitudes of the first inputand the second input, a magnitude of the upward shift in the firstlocation of the first input, and a magnitude of the upward shift in thesecond location of the second input; and wherein manipulating thevirtual object within the virtual environment comprises selecting thevirtual object according to the command in response to the third forcemagnitude exceeding a threshold force magnitude.
 17. The system of claim14: wherein transforming the first input and the second input into thegesture comprises generating a grasping gesture based on coaxialproximity of the first location and the second location through thehandheld device; wherein assigning the magnitude to the gesturecomprises writing a grasping magnitude related to a sum of the firstforce magnitude and the second force magnitude to the gesture; andwherein manipulating the virtual object within the virtual environmentcomprises coupling a virtual tool to the virtual object according to thegrasping magnitude of the gesture.
 18. The system of claim 17: furthercomprising tracking a position of the handheld device within a realspace; wherein manipulating the virtual object within the virtualenvironment comprises: moving the virtual object and the virtual toolwithin the virtual environment according to a change in the position ofthe handheld device within the real space while the sum of the firstforce magnitude of the first input and the second force magnitude of thesecond input exceeds an upper threshold force magnitude; decoupling thevirtual tool from the virtual object within the virtual environment inresponse to the sum of the first force magnitude of the first input andthe second force magnitude of the second input falling below a lowerthreshold force magnitude; and slipping the virtual object relative tothe virtual tool within the virtual environment according to a frictionmodel in response to the sum of the first force magnitude of the firstinput and the second force magnitude of the second input falling withina force magnitude range between the upper threshold force magnitude andthe lower threshold force magnitude.
 19. The system of claim 14, whereinextracting the first force magnitude and the first location of the firstinput and the second force magnitude and the second location of thesecond input comprise: identifying a set of discrete input areas in thetouch image; and labeling each discrete input area in the set ofdiscrete input areas as one of a palm, a first digit, a second digit, athird digit, a fourth digit, and a fifth digit of a hand; whereintransforming the first input and the second input into the gesturecomprises generating a gesture defining adduction and abductionpositions of each virtual finger of a virtual hand model based onlocations and labels of each discrete input area in the set of discreteinput areas; wherein assigning the magnitude to the gesture comprisesaugmenting the gesture with a magnitude of flexion of each virtualfinger of the virtual hand model based on a force magnitude of eachdiscrete input area in the set of discrete input areas; and whereinmanipulating the virtual object within the virtual environment comprisestransforming the virtual hand model in the virtual environment accordingto the adduction and abduction positions and magnitudes of flexiondefined in the gesture.
 20. The system of claim 14, wherein transformingthe first input and the second input into the gesture comprisestransforming the first input and the second input into the gesture of atype corresponding to a sleeve overlay transiently installed over thehandheld device.
 21. The system of claim 14, further comprisingextracting a third force magnitude of a third input at a third locationon the second side of the handheld device from the touch image; whereintransforming the first input and the second input into the gesturecomprises generating a bending gesture according to lateral separationbetween the second location and the third location on the second side ofthe handheld device spanning the first location on the first side of thehandheld device; wherein assigning the magnitude to the gesturecomprises assigning a magnitude of a bending moment to the gesture basedon the first force magnitude, the second force magnitude, and the thirdforce magnitude; and wherein manipulating the virtual object within thevirtual environment comprises virtually bending the virtual objectaccording to the bending moment assigned to the gesture.